CI pipeline is an essential part of modern software project and one of the most popular CI-servers is Jenkins. Managing and setting up a highly available Jenkins server can be a tedious job. For smaller projects, cost of running a multi-instance setup is often too expensive.
If you are willing to accept minor downtime in case of a sudden server loss, we demonstrate here a simple self-healing single server setup that you can use in your AWS environment.
The setup uses AWS EBS. EBS is highly available and persistent block storage service, that we can use to persist Jenkins state between server terminations.
We use AWS autoscaling group with node count of exactly one that automatically restarts new server if the existing server is suddenly terminated (remember, in reality everything fails, all the time ).
We use AWS Route53 for registering a domain for the newly created Jenkins instance – so that access to the server is not bound to node IP address that changes upon termination.
To glue everything together we use EC2 instances user data and AWS Cloudformation for documented and repeatable setup.
To use this template you should have basic VPC with at least one public subnet and Route53 hosted zone already set up.
This setup only focuses on providing bare minimal setup for durable Jenkins server. It does not take into account added features that you should use in production e.g. how you can do SSL termination ( you can use AWS ALB or ELB in front of this Jenkins to terminate the SSL connection.)
Simple jenkins setup architecture diagram
For the sake of simplicity we install the Jenkins into public subnet, but in production usage you should run it in private subnet and manage the network access depending on your networking configuration.
First your EC2 instance needs some permissions to call AWS API. We define IAM role that has policy that grants access to listing and attaching EBS volumes and allows EC2 to update its own DNS record in the given Route53.
JenkinsIamRole:
Type:
AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version:
2012
-10
-17
Statement:
-
Effect:
Allow
Principal:
Service:
-
ec2.amazonaws.com
Action:
- sts:
AssumeRole
Policies:
-
PolicyName:
Attach-Volume-and-Update-Route53-AND-S3
PolicyDocument:
Version:
2012
-10
-17
Statement:
-
Effect:
Allow
Action:
- route53:
ChangeResourceRecordSets
Resource:
!Sub
"arn:aws:route53:::hostedzone/${Route53HostedZoneId}"
-
Effect:
Allow
Action:
- route53:
GetHostedZone
- route53:
ListHostedZones
Resource:
"*"
-
Effect:
Allow
Action:
- ec2:
AttachVolume
- ec2:
DescribeVolumeAttribute
- ec2:
DescribeVolumeStatus
- ec2:
DescribeVolumes
Resource:
"*"
RoleName:
jenkins-role
The real magic happens in the userdata script that is executed when the EC2 instance is started. When the EC2 instance is first time started a custom user data script is run. This configures the instance with defined software packages and it is also the place to run you EBS mounting and Route53 registration code.
# Volume /dev/sdh (which will get created as /dev/xvdh on Amazon Linux)
MOUNT_STATE= "unknown"
until [ " ${!MOUNT_STATE}
"
== "attached"
]; do
MOUNT_STATE=$(aws ec2 describe-volumes \
--region ${AWS::Region}
\
--filters \
Name=attachment.instance-id,Values= ${!INSTANCE_ID}
\
Name=attachment.device,Values=/dev/sdh \
--query Volumes[].Attachments[].State \
--output text)
sleep 5 done
# Format /dev/xvdh if it does not contain a partition yet
if
[ " $(file -b -s /dev/xvdh)
"
== "data"
]; then
mkfs -t ext4 /dev/xvdh fi
mkdir -p /var/lib/jenkins
mount /dev/xvdh /var/lib/jenkins # Persist the volume in /etc/fstab so it gets mounted again
echo
'/dev/xvdh /var/lib/jenkins ext4 defaults,nofail 0 2'
>> /etc/fstab
When the Jenkins is up and running we need to bind it to some domain name (in this example jenkins.{yourdomain}) , so you don’t have to lookup the IP address every time the server restarts.
# Register jenkins to DNS
HOSTED_ZONE_ID= ${HOSTED_ZONE_ID}
DOMAIN=$(aws route53 get-hosted-zone --id " $HOSTED_ZONE_ID
"
--query "HostedZone.Name"
--output text)
PUBLIC_IP=$( curl https://169.254.169.254/latest/meta-data/public-ipv4 ) echo
"Registering $IP
to hostedzone $HOSTED_ZONE_ID
as jenkins. ${!DOMAIN}
"
cat << EOF >> update-route53.json
{ "Comment"
: "Update Jenkins DNS to match IP of new EC2 instance"
, "Changes"
: [
{ "Action"
: "UPSERT"
, "ResourceRecordSet"
: { "Name"
: "jenkins. ${!DOMAIN}
"
, "Type"
: "A"
, "TTL"
: 300, "ResourceRecords"
: [{ "Value"
: " $PUBLIC_IP
"
}]
}
}
]
}
EOF
aws route53 change-resource-record-sets --hosted-zone-id " $HOSTED_ZONE_ID
"
--change-batch file://update-route53.json
To gain access to Jenkins for the first time you need get the first time login password from Jenkins.
if
[ -e /var/lib/jenkins/secrets/initialAdminPassword ]; then
JENKINS_PASSWORD=$(cat /var/lib/jenkins/secrets/initialAdminPassword) echo
"Your Jenkins Password $JENKINS_PASSWORD
"
else
echo
"No initial password Jenkins has allready been initialized"
fi
This part of script checks if this is first Jenkins start and prints the first time login password into std-out. You can view the userdata scripts output in AWS Console.
Go to AWS console -> ec2 -> select jenkins instance
In the context menu select Instance settings -> Get system log
and you should see in the bottom part of the log a entry Your Jenkins Password **********
. You can copy paste that password into Jenkins ui and login.
Now you have successfully created a simple and durable Jenkins setup =).
You can view/download the whole template jenkins.yaml here