CICD with Jenkins and AWS

CICD_jenkins_aws

Read on how to create a CI/CD pipeline from scratch using Jenkins!

Note: Before we start cloning repos if we get permission denied, we can enter the commands below to fix it:

Create a SSH connection from localhost to GitHub

Generate SSH key on localhost

cd into ~/.ssh
enter the command ssh-keygen -t rsa -b 4096 -C "email@email.com"
Follow the instructions as seen in the image below:

Put a lock on GitHub with ssh – copy the public SSH to GitHub

Enter cat 103a.pub to view your ssh key. Copy and paste all of it onto github as shown on the image below:

Creating a new repo for CICD on GitHub

Save the key and create a new repo and select 'SSH' after creating it, as seen below:

Go back to your terminal and create a new directory and add a README.md:

Go back to your repo and enter the instructions into your terminal:

After entering all the commands, if you get an error that says that you don't have permissions, enter the code below:

GitHub documentation on how this works can be read here.

Starting with Jenkins

Open your Jenkins link and login using the details you have been given.

On the homepage select new item or build and then enter an item name:

In our case we are building two items, one to start a test build to see the date and time of the build and the other build is the post build, for the first test.

Build 1 is called yacob_jenkins_test: it's configure contents are as seen below:

This to call the date and time:

This is to start the test_2 post build:

Build 2:

The ps aux command is a tool to monitor processes running on your Linux system:

build time and date from yacob_jenkins_test:

Select console output under the build number and you'll see a separate page saying the build was a success and what the post build is as seen below:

Click on test_2 and click on it's console output and you'll see that the ps aux command was successfully executed:

Setting up Jenkins to create a CI/CD pipeline

Generate a new key of cloned repo you want to push

ssh-keygen -t rsa -b 4096 -C "email@email.com"
enter eval "$(ssh-agent -s)" and ssh-add ~/.ssh/103a to give yourself permissions to commit the cloned repo.
do cat yourkey.pub to obtain the public key which we'll put on github.

Copying the key to github

Enter your cloned repo, select the settings tab INSIDE the repo.
Deploy keys > Add deploy key
Copy the public key you copied and enter it into the key field, with an appropriate title.

Connecting the repo to Jenkins

{Awaiting screenshot but AWS hosting jenkins is down...}

Setting up webhook

Go to the settings tab in your cloned repo on github > create new github
Enter the payload url of your jenkins server. E.g. `69.594.384.22:8080/github-webhook/
Content type application/json
Leave secret blank
Send me everything for events triggered

Creating three jenkins jobs to create the CI/CD pipeline.

First job to pull and push dev builds

Create new item
Discard old builds, set 3 for max builds to keep
Select GitHub project, enter the HTTP link to your repo
Under office 365 connector, select Restrict where this project can be run and type sparta-ubuntu-node as we want to run it from this node.
Under source code management select the git option.
Enter your SSH repository link under Repository URL
Under credentials, we need to add the private SSH key, we can obtain the private key from our terminal. Paste that under credentials by adding a new key
Under Branches to build enter */dev as we want to push the dev builds
Under Build Triggers select GitHub hook trigger for GITScm polling, this is to ensure that jenkins is notified from the webhook.
Under Build Environment select Provide Node & npm bin/folder to PATH
- Select Add post-build action and click Build other projects, enter the name of your second job (once you have made it)

Second job to merge dev branch to main branch

Create new item
Discard old builds, set 3 for max builds to keep
Select GitHub project, enter the HTTP link to your repo
Under office 365 connector, select Restrict where this project can be run and type sparta-ubuntu-node as we want to run it from this node.
Under source code management select the git option.
Enter your SSH repository link under Repository URL
Under credentials, we can enter the SSH private key that we already have.
Under Build Environment select Provide Node & npm bin/folder to PATH
Under Branches to build enter */dev, we will push to main branch in the next few steps.
Under the build option, select Execute Shell and enter:

git checkout main
git merge origin/dev

Post-build actions

Select Add post-build action > Git Publisher
Click Push Only If Build Succeeds - this action self-explanatory
Select Add post-build action again and click Build other projects, enter the name of your third job (once you have made it)
Ensure that Trigger only if build is stable is selected.

Creating EC2 instance in AWS

A very straightforward process, from the VPCs and its containers, create a new instance, ensuring the ports and inbound rules are open for your own IP and for the jenkins ip.

Creating the third job

Create new item
Discard old builds, set 3 for max builds to keep
Select GitHub project, enter the HTTP link to your repo
Under office 365 connector, select Restrict where this project can be run and type sparta-ubuntu-node as we want to run it from this node.
Under source code management select the git option.
Enter your SSH repository link under Repository URL
Under credentials, we can enter the SSH private key that we already have.
Under Build Environment select Provide Node & npm bin/folder to PATH
Under Branches to build enter */main
Enter your file.pem to be able to enter the EC2 instance
Under build, we can enter:

scp -o "StrictHostKeyChecking=no" -r app ubuntu@IPfromaws:~

Creating a CI/CD pipeline from scratch using Jenkins!

In the following steps, I will show the steps to create a CI/CD pipeline using jenkins.

In short the steps are:

Create a master node and agent node instance on AWS. Create an AMI of the agent node
Install jenkins on the master node and the required plugins
Create webhooks and link github repo to jenkins using private and public ssh keys
Connect AWS credentials to jenkins and set configuration to create agent node instance from AMI to launch on AWS
Create jobs to automate builds and tests.

Step 1

Create a master agent where the jenkins server will be built. This is built using an EC2 instance on AWS.

Create an instance on EC2, choose your subnets and so on. Auto assign Public ip. For ports, follow as shown below. Ensure you remember the security group names as you'll need it later.

At the same time, you'll want to create another EC2 instance for the agent node. You'll take an AMI of this after you're done. The ports will be as seen below:

For the agent node:

SSH into it
run sudo apt update && sudo apt upgrade -y && sudo apt install default-jre -y to update/upgrade and install java
create an AMI, you'll need this later.

For the master node:

SSH into it
run sudo apt update && sudo apt upgrade -y && sudo apt install default-jre -y to update/upgrade and install java
curl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo tee \ /usr/share/keyrings/jenkins-keyring.asc > /dev/null to add they key to our system
echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \ https://pkg.jenkins.io/debian-stable binary/ | sudo tee \ /etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt-get update
sudo apt-get install jenkins
sudo systemctl start jenkins

Open jenkins in your browser on port 8080:

Enter sudo cat /var/lib/jenkins/secrets/initialAdminPassword in your terminal to obtain the password.
Enter the password into the terminal
Choose what option suits you best, the first option will take longer:

Enter your login details

Once you're in, go to plugins:

Go to the 'available' tab and ensure you tick all the plugins you'll need, such as AWS EC2, SSH Agent, Office 365 connector, Github, node.js etc.

Once that is done, click download and restart at the bottom.

Restart Jenkins:

After it restarts:

Once your jenkins restarts, go to Manage Jenkins > Manage Nodes and Clouds

Go to configure clouds

Add new cloud > Amazon EC2

Configure your cloud, add your access and secret keys:

Enter the key from your pem file

Test connection to ensure it is connected and set the correct region.

Next, add AMIs to add your agent nodes:

In the description section, enter the name of your agent node that you want to see as your EC2 instance name and enter the AMI id of the agent node that you took a snapshot of:

Set instance type to T2micro
Enter the security group name of your master node
set /home/ubuntu/ for Remote FS root
set ubuntu for Remote user
under AMI Type set sudo for Root command prefix
set 22 for Remote ssh port
For labels enter the name of your agent (e.g. jenkins-agent)
click on Advanced below Init script
set 1 for Number of Executors
add a Tag with Name for Name and jenkins-agent for Value
Apply and save

Launch your agent node to see if it works:

Go back to your EC2 dashboard and see if the instance has been launched:

Check the agent node logs to see if it has launched in the instance:

Set number of executors in the built in node to 0, so only the agent nodes are running the tests:

Step 2 - Setting up webhook

Go to the github repo where you want to push your changes from.

Go to Settings from your repo > Wehbooks > Add webhook and add your webhook:

Add the public SSH key from your local host to the repo:

Step 3: Making the jobs

Job 1

Build item > Freestyle project > OK

Job 2

Job 3

For job 3, enter the ip of your instance where you want to host your app.

Testing your pipeline

I have written this readme alongside creating the pipeline from scratch so it should work once you push a change from your localhost > repo > jenkins > deployed to AWS.

Sources:

https://pkg.jenkins.io/debian-stable/

https://www.kisphp.com/linux/setup-jenkins-server-on-aws-ec2-with-slave-agents

yfp-c/CICD_jenkins_aws