Building a durable Jenkins CI setup in AWS

Hannu-Pekka Hakamäki • marrask. 24, 2017

CI pipeline is an essential part of modern software project and one of the most popular CI-servers is Jenkins. Managing and setting up a highly available Jenkins server can be a tedious job. For smaller projects, cost of running a multi-instance setup is often too expensive.

If you are willing to accept minor downtime in case of a sudden server loss, we demonstrate here a simple self-healing single server setup that you can use in your AWS environment.

The setup uses AWS EBS. EBS is highly available and persistent block storage service, that we can use to persist Jenkins state between server terminations.

We use AWS autoscaling group with node count of exactly one that automatically restarts new server if the existing server is suddenly terminated (remember, in reality everything fails, all the time ).

We use AWS Route53 for registering a domain for the newly created Jenkins instance – so that access to the server is not bound to node IP address that changes upon termination.

To glue everything together we use EC2 instances user data and AWS Cloudformation for documented and repeatable setup.

The prerequisite

To use this template you should have basic VPC with at least one public subnet and Route53 hosted zone already set up.

This setup only focuses on providing bare minimal setup for durable Jenkins server. It does not take into account added features that you should use in production e.g. how you can do SSL termination ( you can use AWS ALB or ELB in front of this Jenkins to terminate the SSL connection.)

The basic setup

Simple jenkins setup architecture diagram

For the sake of simplicity we install the Jenkins into public subnet, but in production usage you should run it in private subnet and manage the network access depending on your networking configuration.

The permissions

First your EC2 instance needs some permissions to call AWS API. We define IAM role that has policy that grants access to listing and attaching EBS volumes and allows EC2 to update its own DNS record in the given Route53.

  JenkinsIamRole: 
 Type: 
 AWS::IAM::Role 
 Properties: 
 AssumeRolePolicyDocument: 
 Version: 
 2012 
 -10 
 -17 
 Statement: 
 - 
 Effect: 
 Allow 
 Principal: 
 Service: 
 - 
 ec2.amazonaws.com 
 Action: 
 - sts: 
 AssumeRole 
 Policies: 
 - 
 PolicyName: 
 Attach-Volume-and-Update-Route53-AND-S3 
 PolicyDocument: 
 Version: 
 2012 
 -10 
 -17 
 Statement: 
 - 
 Effect: 
 Allow 
 Action: 
 - route53: 
 ChangeResourceRecordSets 
 Resource: 
 !Sub 
 "arn:aws:route53:::hostedzone/${Route53HostedZoneId}" 
 - 
 Effect: 
 Allow 
 Action: 
 - route53: 
 GetHostedZone 
 - route53: 
 ListHostedZones 
 Resource: 
 "*" 
 - 
 Effect: 
 Allow 
 Action: 
 - ec2: 
 AttachVolume 
 - ec2: 
 DescribeVolumeAttribute 
 - ec2: 
 DescribeVolumeStatus 
 - ec2: 
 DescribeVolumes 
 Resource: 
 "*" 
 RoleName: 
 jenkins-role

The EBS Mounting

The real magic happens in the userdata script that is executed when the EC2 instance is started. When the EC2 instance is first time started a custom user data script is run. This configures the instance with defined software packages and it is also the place to run you EBS mounting and Route53 registration code.

  # Volume /dev/sdh (which will get created as /dev/xvdh on Amazon Linux) 
MOUNT_STATE= "unknown" 
until [ " ${!MOUNT_STATE} 
" 
== "attached" 
]; do 
MOUNT_STATE=$(aws ec2 describe-volumes \
                --region ${AWS::Region} 
\
                --filters \
                    Name=attachment.instance-id,Values= ${!INSTANCE_ID} 
\
                    Name=attachment.device,Values=/dev/sdh \
                --query Volumes[].Attachments[].State \
                --output text)

              sleep 5 done 
 # Format /dev/xvdh if it does not contain a partition yet 
 if 
[ " $(file -b -s /dev/xvdh) 
" 
== "data" 
]; then 
mkfs -t ext4 /dev/xvdh fi 
mkdir -p /var/lib/jenkins
            mount /dev/xvdh /var/lib/jenkins # Persist the volume in /etc/fstab so it gets mounted again 
 echo 
 '/dev/xvdh /var/lib/jenkins ext4 defaults,nofail 0 2' 
>> /etc/fstab

This script first queries the EC2 metadata to get the instance id.
Then it calls AWS api to attach the EBS volume to it.
It waits until the volume has been attached and then, if this is first mount, formats it
Finally it mounts the formatted volume into var/lib/jenkins folder where Jenkins will persists its data.

The Route53 registration

When the Jenkins is up and running we need to bind it to some domain name (in this example jenkins.{yourdomain}) , so you don’t have to lookup the IP address every time the server restarts.

  # Register jenkins to DNS 
HOSTED_ZONE_ID= ${HOSTED_ZONE_ID} 
DOMAIN=$(aws route53 get-hosted-zone --id " $HOSTED_ZONE_ID 
" 
--query "HostedZone.Name" 
--output text)
            PUBLIC_IP=$( curl https://169.254.169.254/latest/meta-data/public-ipv4 ) echo 
 "Registering $IP 
to hostedzone $HOSTED_ZONE_ID 
as jenkins. ${!DOMAIN} 
" 
cat << EOF >> update-route53.json
            { "Comment" 
: "Update Jenkins DNS to match IP of new EC2 instance" 
, "Changes" 
: [
                { "Action" 
: "UPSERT" 
, "ResourceRecordSet" 
: { "Name" 
: "jenkins. ${!DOMAIN} 
" 
, "Type" 
: "A" 
, "TTL" 
: 300, "ResourceRecords" 
: [{ "Value" 
: " $PUBLIC_IP 
" 
}]
                  }
                }
              ]
            }
            EOF

            aws route53 change-resource-record-sets --hosted-zone-id " $HOSTED_ZONE_ID 
" 
--change-batch file://update-route53.json

This script first calls AWS api to get the domain name of the given Hosted zone.
Then it resolves its own public IP address via instance metadata and calls Route53 api to register the instance public ip under jenkins.{your domain} domain name.
After the DNS cache is refreshed, you should see the up and running Jenkins in https://jenkins.{your domain}:8080

Doing the first login

To gain access to Jenkins for the first time you need get the first time login password from Jenkins.

  if 
[ -e /var/lib/jenkins/secrets/initialAdminPassword ]; then 
JENKINS_PASSWORD=$(cat /var/lib/jenkins/secrets/initialAdminPassword) echo 
 "Your Jenkins Password $JENKINS_PASSWORD 
" 
 else 
 echo 
 "No initial password Jenkins has allready been initialized" 
 fi

This part of script checks if this is first Jenkins start and prints the first time login password into std-out. You can view the userdata scripts output in AWS Console.

Go to AWS console -> ec2 -> select jenkins instance
In the context menu select Instance settings -> Get system log and you should see in the bottom part of the log a entry Your Jenkins Password ********** . You can copy paste that password into Jenkins ui and login.
Now you have successfully created a simple and durable Jenkins setup =).

You can view/download the whole template jenkins.yaml here

< Vanhempi viesti Uudempi viesti >