Using Jenkins to Automate Source code Changes on our Devops Team tool.
In the previous project 8 we introduced horizontal scalability concept, which allow us to add new Web Servers to our Tooling Website and we successfully deployed a set up with 2 Web Servers and also a Load Balancer to distribute traffic between them. Manual configuration is easy with few number of servers like 2 or 3 but automation of this configuration is neccessary when then number of servers are in hundreds or larger sum. We will be using Jenkins to automate source code management in this project- in this project we are going to utilize Jenkins CI capabilities to make sure that every change made to the source code in GitHub will be automatically be updated to the Tooling Website.
This is the current project architecture
This is how the new website Architecture would be after the automation
Create an AWS EC2 server based on Ubuntu Server 20.04 LTS and name it "Jenkins"
Install JDK (since Jenkins is a Java-based application)
sudo apt update
sudo apt install default-jdk-headless
Install Jenkins
wget -q -O - https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo apt-key add -
sudo sh -c 'echo deb https://pkg.jenkins.io/debian-stable binary/ > \
/etc/apt/sources.list.d/jenkins.list'
sudo apt update
sudo apt-get install jenkins
Check jenkins is up and running:
sudo systemctl status jenkins
Jenkins server uses TCP port 8080, open the inbound rule in your ec2 instance
Setup Jenkins
From your browser access:
http://<Jenkins-Server-Public-IP-Address-or-Public-DNS-Name>:8080
Get the default admin password
Retrieve it from your server:
sudo cat /var/lib/jenkins/secrets/initialAdminPassword
Choose suggested plugins when prompted to choose plugin to install
When the pluggins is done installing; Create an admin user and you will get your Jenkins server address
Installation is complete and ready for use.
In this part, we will configure a simple jenkins job/project. This job will will be triggered by GitHub webhooks and will execute a ‘build’ task to retrieve codes from GitHub and store it locally on Jenkins server.
-
Open the tooling project on git hub
-
Click on settings
-
Click on webhook
-
Enter
http://<jenkins-server-public-ip-address>:8080/github-webhook/ -
Select application/json
Go to Jenkins web console, click "New Item" and create a "Freestyle project"
To connect your GitHub repository, you will need to provide its URL, you can copy from the repository itself
In configuration of the Jenkins freestyle project choose Git repository, provide there the link to your Tooling GitHub repository and credentials (user/password) so Jenkins could access files in the repository.
Save the configuration and let us try to run the build
Click "Build Now" button, if you have configured everything correctly, the build will be successfull and you will see it under #1
You can open the build and check in "Console Output" if it has run successfully
Let us make this project build run automatically since it still runs manually at this stage:
Click "Configure" your job/project and add these two configurations Configure triggering the job from GitHub webhook:
Go ahead and make some change in any file in your GitHub repository (e.g. README.MD file) and push the changes to the master branch.
You will see that a new build has been launched automatically (by webhook) and you can see its results – artifacts, saved on Jenkins server.
By default, the artifacts are stored on Jenkins server locally
ls /var/lib/jenkins/jobs/<Project-tooling_github-name>/builds/<build_number>/archive/
Since we now have the artifacts saved locally in our Jenkins Server, we want to copy them to the NFS SERVER to the /mnt/apps directory
Install Publish Over SSH plugin
On main dashboard select "Manage Jenkins" and choose "Manage Plugins" menu item.
On "Available" tab search for "Publish Over SSH" plugin and install it
Configure the job/project to copy artifacts over to NFS server.
On main dashboard select "Manage Jenkins" and choose "Configure System" menu item.
Scroll down to Publish over SSH plugin configuration section and configure it to be able to connect to your NFS server:
-
Provide a private key (content of .pem file that you use to connect to NFS server via SSH/Putty)
-
A name
-
Hostname – can be private IP address of your NFS server
-
Username – ec2-user (since NFS server is based on EC2 with RHEL 8)
-
Remote directory – /mnt/apps since our Web Servers use it as a mointing point to retrieve files from the NFS server
Ensure to test and get the success message before saving.
Save the configuration
Open your Jenkins job/project configuration page and add another one "Post-build Action"
Configure it to send all files probuced by the build into our previouslys define remote directory. In our case we want to copy all files and directories – so we use **.
Save this configuration and go ahead, change something in README.MD file in your GitHub Tooling repository.
Webhook will trigger a new job and in the "Console Output" of the job you will find something like this:
Connect to your server through the terminal and run
cat /mnt/apps/README.md
If you see exactly the change you made on your READme file, the job has successfully been set up.
End