AWS Jenkins Setup with Autoscaling EC2 Slaves

Written By : Lam Chan

Jenkins is one of the most popular CI build tools out there.  It isn't the best, but it does have a lot of community support with plugins.  My goal was to build out a jenkins master slave cluster that would leverage docker for integration test and containerization of my applications.  My secondary goal was to have the slaves auto scale according to queue size so that compute resources wasn't wasted.

Master Build Out

First, log into your AWS EC2 console and start a new instance with an Amazon Linux AMI 63bit HVM image.  For the configuration, make sure you allow port access for 80, 50000 and 22.  

Update Yum

After the EC2 instance comes up, you'll need to ssh into it.  Once you're in, 

sudo yum update

Install Nginx

We're going to install Nginx so we can proxy request from port 80 to port 8080.  Port 8080 is what Jenkins listens to over http.  You can try to optionally try and setup Jenkins to listen to port 80, but to do that, Jenkins will need root access which is not recommended.  

sudo yum install -y nginx

After, let's make sure nginx is set to start on boot.

chkconfig nginx on

Locate the nginx.conf file so we can add the port forwarding setup.  This file is located in one of the following places (/usr/local/nginx/conf, /etc/nginx, /usr/local/etc/nginx)

Modify the file with the following additions

server {
    listen 80;

    location / {
        # Backend nodejs server
        proxy_pass          http://127.0.0.1:8080;
        proxy_http_version  1.1;
        proxy_set_header    Upgrade     $http_upgrade;
        proxy_set_header    Connection  $connection_upgrade;
    }
}

Install Jenkins

Use the following commands in sequence.

sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo
sudo rpm --import http://pkg.jenkins-ci.org/redhat-stable/jenkins-ci.org.key
yum install -y jenkins

Let's make sure jenkins starts on boot

chkconfig jenkins on

Install Docker

For all of our Jenkins builds, I wanted to use docker for integraiton test and containerization for Elasticbeanstalk delivery.  

yum install -y docker

After, we'll need to add users to the docker group so we don't need to sudo call docker due to the docker.run gating.

sudo usermod -a -G docker ec2-user
sudo usermod -a -G docker jenkins

Reboot

After all of that, reboot so the permissions take place for docker.

sudo reboot

Once the machine comes back up, try hitting the ec2 public dns and Jenkins should come up.

Slave Build Out

For this, we need to build out a custom AMI that we can resuse for through the Jenkins EC2 Slave provisioning.  To do this, we need to boot up an EC2 instance like we did earlier, add some dependencies and save it as an AMI image.

First, log into your AWS EC2 console and start a new instance with an Amazon Linux AMI 63bit HVM image.  For the configuration, make sure you allow port access for 22.  

Install Docker

yum install -y docker

After, we'll need to add users to the docker group so we don't need to sudo call docker due to the docker.run gating.

sudo usermod -a -G docker ec2-user

Create and Set Permissions for Build Folder

We need to create and set permissions for the build folder to be the same as what you will enter ("ec2-user") in the configuration panel for the Jenkins EC2 plugin.

mkdir -p /var/lib/jenkins
chmod -R ec2user:ec2user /var/lib/jenkins

Export to an image.

In the AWS EC2 console, select this instance, and for the "Action" drop down, find the "Create Image" button.  After this is done, you'll find your new AMI in the AMI page on the left rail of the screen.

Setup Jenkins with the EC2 Plugin

The rest of the configuration for this plugin can be found here.

Tags: , , , , ,

No comments yet.

Post your comment