Building a durable Jenkins CI setup in AWS
CI pipeline is an essential part of modern software project and one of the most popular CI-servers is Jenkins. Managing and setting up a highly available Jenkins server can be a tedious job. For smaller projects, cost of running a multi-instance setup is often too expensive.
If you are willing to accept minor downtime in case of a sudden server loss, we demonstrate here a simple self-healing single server setup that you can use in your AWS environment.
The setup uses AWS EBS. EBS is highly available and persistent block storage service, that we can use to persist Jenkins state between server terminations.
We use AWS autoscaling group with node count of exactly one that automatically restarts new server if the existing server is suddenly terminated (remember, in reality everything fails, all the time ).
We use AWS Route53 for registering a domain for the newly created Jenkins instance – so that access to the server is not bound to node IP address that changes upon termination.
To glue everything together we use EC2 instances user data and AWS Cloudformation for documented and repeatable setup.
The prerequisite
To use this template you should have basic VPC with at least one public subnet and Route53 hosted zone already set up.
This setup only focuses on providing bare minimal setup for durable Jenkins server. It does not take into account added features that you should use in production e.g. how you can do SSL termination ( you can use AWS ALB or ELB in front of this Jenkins to terminate the SSL connection.)
The basic setup
Simple jenkins setup architecture diagram
For the sake of simplicity we install the Jenkins into public subnet, but in production usage you should run it in private subnet and manage the network access depending on your networking configuration.
The permissions
First your EC2 instance needs some permissions to call AWS API. We define IAM role that has policy that grants access to listing and attaching EBS volumes and allows EC2 to update its own DNS record in the given Route53.
JenkinsIamRole:
Type:
AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version:
2012
-10
-17
Statement:
-
Effect:
Allow
Principal:
Service:
-
ec2.amazonaws.com
Action:
- sts:
AssumeRole
Policies:
-
PolicyName:
Attach-Volume-and-Update-Route53-AND-S3
PolicyDocument:
Version:
2012
-10
-17
Statement:
-
Effect:
Allow
Action:
- route53:
ChangeResourceRecordSets
Resource:
!Sub
"arn:aws:route53:::hostedzone/${Route53HostedZoneId}"
-
Effect:
Allow
Action:
- route53:
GetHostedZone
- route53:
ListHostedZones
Resource:
"*"
-
Effect:
Allow
Action:
- ec2:
AttachVolume
- ec2:
DescribeVolumeAttribute
- ec2:
DescribeVolumeStatus
- ec2:
DescribeVolumes
Resource:
"*"
RoleName:
jenkins-role
The EBS Mounting
The real magic happens in the userdata script that is executed when the EC2 instance is started. When the EC2 instance is first time started a custom user data script is run. This configures the instance with defined software packages and it is also the place to run you EBS mounting and Route53 registration code.
# Volume /dev/sdh (which will get created as /dev/xvdh on Amazon Linux)
MOUNT_STATE= "unknown"
until [ " ${!MOUNT_STATE}
"
== "attached"
]; do
MOUNT_STATE=$(aws ec2 describe-volumes \
--region ${AWS::Region}
\
--filters \
Name=attachment.instance-id,Values= ${!INSTANCE_ID}
\
Name=attachment.device,Values=/dev/sdh \
--query Volumes[].Attachments[].State \
--output text)
sleep 5 done
# Format /dev/xvdh if it does not contain a partition yet
if
[ " $(file -b -s /dev/xvdh)
"
== "data"
]; then
mkfs -t ext4 /dev/xvdh fi
mkdir -p /var/lib/jenkins
mount /dev/xvdh /var/lib/jenkins # Persist the volume in /etc/fstab so it gets mounted again
echo
'/dev/xvdh /var/lib/jenkins ext4 defaults,nofail 0 2'
>> /etc/fstab
- This script first queries the EC2 metadata to get the instance id.
- Then it calls AWS api to attach the EBS volume to it.
- It waits until the volume has been attached and then, if this is first mount, formats it
- Finally it mounts the formatted volume into var/lib/jenkins folder where Jenkins will persists its data.
The Route53 registration
When the Jenkins is up and running we need to bind it to some domain name (in this example jenkins.{yourdomain}) , so you don’t have to lookup the IP address every time the server restarts.
# Register jenkins to DNS
HOSTED_ZONE_ID= ${HOSTED_ZONE_ID}
DOMAIN=$(aws route53 get-hosted-zone --id " $HOSTED_ZONE_ID
"
--query "HostedZone.Name"
--output text)
PUBLIC_IP=$( curl https://169.254.169.254/latest/meta-data/public-ipv4 ) echo
"Registering $IP
to hostedzone $HOSTED_ZONE_ID
as jenkins. ${!DOMAIN}
"
cat << EOF >> update-route53.json
{ "Comment"
: "Update Jenkins DNS to match IP of new EC2 instance"
, "Changes"
: [
{ "Action"
: "UPSERT"
, "ResourceRecordSet"
: { "Name"
: "jenkins. ${!DOMAIN}
"
, "Type"
: "A"
, "TTL"
: 300, "ResourceRecords"
: [{ "Value"
: " $PUBLIC_IP
"
}]
}
}
]
}
EOF
aws route53 change-resource-record-sets --hosted-zone-id " $HOSTED_ZONE_ID
"
--change-batch file://update-route53.json
- This script first calls AWS api to get the domain name of the given Hosted zone.
- Then it resolves its own public IP address via instance metadata and calls Route53 api to register the instance public ip under jenkins.{your domain} domain name.
- After the DNS cache is refreshed, you should see the up and running Jenkins in https://jenkins.{your domain}:8080
Doing the first login
To gain access to Jenkins for the first time you need get the first time login password from Jenkins.
if
[ -e /var/lib/jenkins/secrets/initialAdminPassword ]; then
JENKINS_PASSWORD=$(cat /var/lib/jenkins/secrets/initialAdminPassword) echo
"Your Jenkins Password $JENKINS_PASSWORD
"
else
echo
"No initial password Jenkins has allready been initialized"
fi
This part of script checks if this is first Jenkins start and prints the first time login password into std-out. You can view the userdata scripts output in AWS Console.
Go to AWS console -> ec2 -> select jenkins instance
In the context menu select Instance settings -> Get system log
and you should see in the bottom part of the log a entry Your Jenkins Password **********
. You can copy paste that password into Jenkins ui and login.
Now you have successfully created a simple and durable Jenkins setup =).
You can view/download the whole template jenkins.yaml here
Viimeisimmät kirjoitukset
