Building a durable Jenkins CI setup in AWS

Building a durable Jenkins CI setup in AWS

CI pipeline is an essential part of modern software project and one of the most popular CI-servers is Jenkins. Managing and setting up a highly available Jenkins server can be a tedious job. For smaller projects, cost of running a multi-instance setup is often too expensive.

If you are willing to accept minor downtime in case of a sudden server loss, we demonstrate here a simple self-healing single server setup that you can use in your AWS environment.

The setup uses AWS EBS. EBS is highly available and persistent block storage service, that we can use to persist Jenkins state between server terminations.

We use AWS autoscaling group with node count of exactly one that automatically restarts new server if the existing server is suddenly terminated (remember, in reality everything fails, all the time ).

We use AWS Route53 for registering a domain for the newly created Jenkins instance – so that access to the server is not bound to node IP address that changes upon termination.

To glue everything together we use EC2 instances user data and AWS Cloudformation for documented and repeatable setup.

The prerequisite

To use this template you should have basic VPC with at least one public subnet and Route53 hosted zone already set up.

This setup only focuses on providing bare minimal setup for durable Jenkins server. It does not take into account added features that you should use in production e.g. how you can do SSL termination ( you can use AWS ALB or ELB in front of this Jenkins to terminate the SSL connection.)

The basic setup

Simple jenkins setup architecture diagram

Simple jenkins setup architecture diagram

For the sake of simplicity we install the Jenkins into public subnet, but in production usage you should run it in private subnet and manage the network access depending on your networking configuration.

The permissions

First your EC2 instance needs some permissions to call AWS API. We define IAM role that has policy that grants access to listing and attaching EBS volumes and allows EC2 to update its own DNS record in the given Route53.


JenkinsIamRole:
  Type: AWS::IAM::Role
  Properties:
    AssumeRolePolicyDocument:
      Version: 2012-10-17
      Statement:
      -
        Effect: Allow
        Principal:
          Service:
            - ec2.amazonaws.com
        Action:
          - sts:AssumeRole
    Policies:
    -
      PolicyName: Attach-Volume-and-Update-Route53-AND-S3
      PolicyDocument:
       Version: 2012-10-17
       Statement:
       -
         Effect: Allow
         Action:
         - route53:ChangeResourceRecordSets
         Resource: !Sub "arn:aws:route53:::hostedzone/${Route53HostedZoneId}"
       -
         Effect: Allow
         Action:
          - route53:GetHostedZone
          - route53:ListHostedZones
         Resource: "*"
       -
         Effect: Allow
         Action:
         - ec2:AttachVolume
         - ec2:DescribeVolumeAttribute
         - ec2:DescribeVolumeStatus
         - ec2:DescribeVolumes
         Resource: "*"
    RoleName: jenkins-role
          

The EBS Mounting

The real magic happens in the userdata script that is executed when the EC2 instance is started. When the EC2 instance is first time started a custom user data script is run. This configures the instance with defined software packages and it is also the place to run you EBS mounting and Route53 registration code.


            # Volume /dev/sdh (which will get created as /dev/xvdh on Amazon Linux)
            MOUNT_STATE="unknown"
            until [ "${!MOUNT_STATE}" == "attached" ]; do
              MOUNT_STATE=$(aws ec2 describe-volumes \
                --region ${AWS::Region} \
                --filters \
                    Name=attachment.instance-id,Values=${!INSTANCE_ID} \
                    Name=attachment.device,Values=/dev/sdh \
                --query Volumes[].Attachments[].State \
                --output text)

              sleep 5
            done

            # Format /dev/xvdh if it does not contain a partition yet
            if [ "$(file -b -s /dev/xvdh)" == "data" ]; then
              mkfs -t ext4 /dev/xvdh
            fi

            mkdir -p /var/lib/jenkins
            mount /dev/xvdh /var/lib/jenkins

            # Persist the volume in /etc/fstab so it gets mounted again
            echo '/dev/xvdh /var/lib/jenkins ext4 defaults,nofail 0 2' >> /etc/fstab
            
  1. This script first queries the EC2 metadata to get the instance id.
  2. Then it calls AWS api to attach the EBS volume to it.
  3. It waits until the volume has been attached and then, if this is first mount, formats it
  4. Finally it mounts the formatted volume into var/lib/jenkins folder where Jenkins will persists its data.

The Route53 registration

When the Jenkins is up and running we need to bind it to some domain name (in this example jenkins.{yourdomain}), so you don’t have to lookup the IP address every time the server restarts.


            # Register jenkins to DNS
            HOSTED_ZONE_ID=${HOSTED_ZONE_ID}
            DOMAIN=$(aws route53 get-hosted-zone --id "$HOSTED_ZONE_ID" --query "HostedZone.Name"  --output text)
            PUBLIC_IP=$( curl http://169.254.169.254/latest/meta-data/public-ipv4 )
            echo "Registering $IP to hostedzone $HOSTED_ZONE_ID as jenkins.${!DOMAIN}"
            cat << EOF >> update-route53.json
            {
              "Comment": "Update Jenkins DNS to match IP of new EC2 instance",
              "Changes": [
                {
                  "Action": "UPSERT",
                  "ResourceRecordSet": {
                    "Name" : "jenkins.${!DOMAIN}",
                    "Type": "A",
                    "TTL": 300,
                    "ResourceRecords": [{ "Value": "$PUBLIC_IP" }]
                  }
                }
              ]
            }
            EOF

            aws route53 change-resource-record-sets --hosted-zone-id "$HOSTED_ZONE_ID" --change-batch file://update-route53.json
                
  1. This script first calls AWS api to get the domain name of the given Hosted zone.
  2. Then it resolves its own public IP address via instance metadata and calls Route53 api to register the instance public ip under jenkins.{your domain} domain name.
  3. After the DNS cache is refreshed, you should see the up and running Jenkins in http://jenkins.{your domain}:8080

Doing the first login

To gain access to Jenkins for the first time you need get the first time login password from Jenkins.


            if [ -e /var/lib/jenkins/secrets/initialAdminPassword ]; then
              JENKINS_PASSWORD=$(cat /var/lib/jenkins/secrets/initialAdminPassword)
              echo "Your Jenkins Password $JENKINS_PASSWORD"
            else
              echo "No initial password Jenkins has allready been initialized"
            fi
                

This part of script checks if this is first Jenkins start and prints the first time login password into std-out. You can view the userdata scripts output in AWS Console.

Building a durable Jenkins CI setup in AWS

Go to AWS console -> ec2 -> select jenkins instance 
In the context menu select Instance settings -> Get system log and you should see in the bottom part of the log a entry Your Jenkins Password **********. You can copy paste that password into Jenkins ui and login.
Now you have successfully created a simple and durable Jenkins setup =).

You can view/download the whole template jenkins.yaml here

Senior Cloud Architect, Partner

1 kommentti

  1. Antonio23.07.2018

    the user_data script is incomplete. the point 1 and 2 are missing.

    Vastaa

Vastaa

Sähköpostiosoitettasi ei julkaista. Pakolliset kentät on merkitty *