Summary

Michael & Sketch assignment solution repository.

This repository will have all the code and documentation needed for the proposed exercises.

Describing the solution (ex. 1)

The target of this solution is to have a tiny CI/CD environment for the mgtascu/guava open-source project which was previously forked from google/guava.

Approach

This solution is oriented on EaC(everything as code). Some of the benefits from having EaC are:

Reducing human intervention which is error prone
A better solution management (versioning, portability, security, etc)
Consistency and scalability

Tech stack

The underlying environment for the Jenkins server is cloud-based (AWS) which is provisioned through terraform IaC.

The Jenkins web-server is running inside a docker container, making it fast to install, running in an isolated environment, thus easy to replicate on another infrastructure. The configuration, plugins and the job are all generated by code, with little to none human interaction, with the help of DevOps tools like JCasc, JobDSL, GroovyDSL, Jenkins Job Builder.

guava is a java with maven project, which usually means there are a lot of dependencies when running the build. Mounting the docker daemon into the Jenkins container, enables us to run the build itself in a separate container, prepared to execute the maven lifecycle of this build, therefore skipping the need to prepare an environment for the run.

The GitHub status checks are easy to install through the UI and for the purpose of this exercise I did so. However, there is are ways to automatically set them up, by making use of the graphql calls to the github's api, or by using terraform's github provider

Note: to run the solution as smoothly as possible, the configurability of the execution is reduced considerably, aiming to lessen the data needed.

Prerequisites

Assuming there is an AWS account available, here is a list of requirements to run the solution.

An environment with awscli and terraform installed. While developing the solution, I used the a t2.micro AWS EC2 instance running Amazon Linux 2 and the following tool versions.

aws-cli/1.18.147 Python/2.7.18 Linux/5.10.109-104.500.amzn2.x86_64 botocore/1.18.6 or aws-cli/2.2.31 Python/3.8.8 Darwin/21.4.0 exe/x86_64 prompt/off

Terraform v1.1.9
Subscribe to the AWS Marketplace Centos AMI, if you haven't already. The terraform apply command will require this step and fail otherwise.
The awscli must be configured with the credentials of an IAM user with the proper access to create and delete the required infra(sg, key pairs and instances, in ec2). To achieve this, run aws configure and input your user details.

AWS Access Key ID []: <your_user_access_key_id>

AWS Secret Access Key []: <your_user_secret_access_key>

Default region name []: eu-west-1

Default output format []: json

Note: The ami-id used in the solution belongs to eu-west-1(Ireland) region. Please use the same region, to avoid searching for a matching ami in a different region.
Place the solution repository to a location of your choosing in the environment and switch to that location. I will refer to it as /tmp/solution in the following section.

Run book

In this section we go through a series of steps to successfully run the solution.

To create and execute a simple build pipeline for the guava repository, firstly we need an environment with this capability. We will be using terraform to provision a landing zone, composed of a security group, a ssh key pair and an EC2 t3.medium instance.

Note: A github token with minimal access was created for this exercise. For security reasons, it cannot be pushed to the remote repository. Please find it in the demo_token.txt attached file.

Creating the landing zone

Change the working directory to tfcode

cd /tmp/solution/tfcode
Initializing the terraform working directory containing the terraform configuration files

terraform init
Create a terraform plan to visualize the set of changes which will be applied to your AWS account

terraform plan -var="github_token=<the_additionally_shared_token>"
Apply the previously planned changes to create the landing zone automatically

terraform apply -var="github_token=<the_additionally_shared_token>"

This operation will take a few minutes

After the apply command has finished running, there will be an output printed to the console with some useful data, similar to the one below such as the Jenkins access URL(jenkins_server_url) and the generated ssh key to access the Jenkins server (ssh_key_name):

   Apply complete! Resources: 5 added, 0 changed, 0 destroyed.
   Outputs:
   ec2_id = "i-02842e6e3b2188592"
   ec2_private_ip = "172.31.23.109"
   ec2_public_ip = "54.217.178.253"
   jenkins_server_url = "http://ec2-54-217-178-253.eu-west-1.compute.amazonaws.com:8080"
   ssh_key_name = "jenkins_ssh_access_key.pem"

Running the pipeline

Now that the landing zone is create, let us run the solution

Copy the jenkins_server_url from the console, and paste it into a browser

You will be prompted with a Jenkins login

Login Credentials:
- username: user
- password: userpass

Once logged in, you will find a seed job waiting for you :). Use this job to generate the pipeline for the guava repository, by building it straight away. To do so, click on the job name and the on Build Now
After this build completed, the guava repository build pipeline is created and already running the scan for existing PRs
In the fork that I have prepared for this exercise, there are 2 PRs that are matching the scan criteria, thus they are discovered and built right away. A run consists of 3 stages simple stages, like in the below screenshot
After the builds succeeded, the build artifacts are available on the build page to easily download them

Completing the exercise

To shell case completing all the challenges of this exercise, I'll show the GitHub integration side in this section.

Branch Protection Rule
- On the repository's settings page, under branches, I have added a branch protection rule for the release/* branch name pattern.
- The rule currently applies to 3 branches and require the continuous-integration/jenkins/pr-head status check to pass before unlocking the pull request merge into the matching branches.
PR Status Checks
- In one of the prepared PRs there is a failure, introduced on purpose, to block the merge due to status check failing.
- The other PR however, has no issue at build time, therefore it passes the status check and allows the merge to happen.

Clearing the environment

To avoid generating redundant cost, after the execution has solved its purpose, the environment can be destroyed easily from where it was created.

cd /tmp/solution/tfcode
terraform destroy -var="github_token=<the_additionally_shared_token>"

Rolling out to production (ex. 2)

While the solution presented in the first part of the documentation is quite robust, it is not designed to run continuous hours of operation nor to scale it to multiple pipelines. To bring it to a production level environment, we need to build more around this prototype.

Strategy

Assuming AWS as the infrastructure provider,

Assuming guava as the main content in this production environment :D

Infrastructure

The first area to bring into attention for a production ready environment, would be to scale the infrastructure. Making use of the Auto-Scaling service and the Load balancer service in AWS, we can enable Jenkins to use more resources when needed while maintaining a cost efficient approach. In doing so, we can afford using workers for Jenkins, and avoid running any builds on the master.

Another very important aspect of the production environment would be to have a disaster recovery mechanism, ensuring zero-downtime(or closet to) of the environment. We can leverage terraform infrastructure as code, to replicate the environment in a different region. On the configuration side, having the server, plugins and jobs as code, will enable us to be up an running quickly.

Considering a cloud-native solution, and the 2 areas above, an ideal production infrastructure for Jenkins server would be to have the containerized Jenkins run in the control plane of AWSs managed kubernetes (EKS), while managing the workload on the node groups attached to the cluster. The node groups would be making use of auto-scaling to provision infrastructure dynamically and the Load balancer will offer access to the server. Jenkins will launch the workers inside pods and they can be tailored to the pipeline needs.

Pipeline

On the CI side of the pipeline, to achieve a production ready pipeline for the guava project, we can use a set of jobs to cover the development needs. In addition, an artifact repository manager would be used to store the build artifacts(ex. Artifactory, Nexus).

Considering the GitHub branching strategy and the matrix of configuration, we have the following scenarios:

CI (Continuous Integration) build
- Builds the main/master branch, the current line of development. Always has the latest working code.
- The lifecycle of this branch is permanent, as long as the repository exists.
- PRs to this branch can only be merged when both main and head branch are green.
- Builds the code, runs the tests and pushes the artifacts to the "stable/release" repository.
Release build
- Builds the release/* branches. Branched form main/master at a point in time, when releasing a version of the artifacts. Mainly used to address patches and hot-fixes for the released version.
- The lifecycle of this branch ends when the support for the released version ends.
- PRs to this branch can only be merged when both release and head branch are green.
- Builds the code, runs the tests and pushes the artifacts to the "stable/release" repository.
PF (Pre-Flight/Pre-Commit) build
- Builds the feature/private branches. Usually created by 1 developer to address a code change/feature implementation
- The lifecycle of this branch stops when it is merged
- Builds the code quickly, by default skipping the tests run, but with an option to enable it if desired. Pushes the artifacts to the "sandbox/snapshot" repository.
PR (Pull-Request) build
- Builds a merge commit between the base and the head branch.
- There is no branch associated with this. It exists until the PR is merged or closed.
- Builds the code, runs the tests and does not push any artifacts.
- Can be leveraged to validate if the merge breaks the CI/Release and for automating the merge.

For the CD side of the pipeline there isn't much to do, since the guava project is a core library for java. However, assuming we have an automated way to deploy and validate one or more use cases of this project, we can extend the CI and Release pipelines to include this part before promoting an artifact.

Considerations

When implementing a production environment, a major consideration goes to the security factor. In order to address the security concerns about the infrastructure, we can apply AWSs best practices. The access to the builds can be role-based controlled by leveraging Jenkins plugins and the access to the Jenkins server can be restricted to allow only connections via a dedicated VPN.

Cost mitigation is crucial when building production or scaled environments. Keeping it in check from early on is a great idea, making it harder to snowball out of control and inducing downtime for maintenance and refactoring.

When building artifacts, versioning plays an important role in organizing and orchestrating a healthy CI/CD environment. Using versioning patters such as semantic versioning can prove to be quite beneficial in the long run.

An easier alternative

Many features can boil down to having the required resources at a point in time. Using a cutting-edge tech stack sometimes introduce complexity and can delay things in a time of need. If time is of the essence when building a production environment like this one, a fixed infrastructure can be deployed using on-demand or reserved instances. The Jenkins server will be hosted on a node by itself and a couple of other instances will be connected to the master as permanent nodes. With containerized builds, the pipeline side won't change much form the initial approach and the solution is still robust, but things like high availability, scalability and cost optimization will be traded off.

Introducing XCode build (ex. 3)

To integrate an iOS build in our Jenkins server we need to enable it to build on iOS using XCode tools. After some research, it looks like there is a Jenkins plugin to handle XCode or we can run the commands from shell directly if we need to avoid using this plugin. On the system side, we need a server running macOS to connect is as a permanent node to our Jenkins server.

The pipeline will run on this node only(at least for the build part) and have the same set of jobs associated. The artifacts can be pushed to our artifact repository manager and then consumed in the other project. If there is a dependency to build the XCode project every time before running a build, we can setup a project dependency relationship using a Jenkins plugin or api calls.

mgtascu/solution