Jenkins CICD Docker Pipeline Using Ephemeral Slaves and HTTPS

This repository contains custom Docker files for running Jenkins using ephemeral build slaves over a NGINX reverse proxy. Everything is setup to run on HTTPS using a self-signed certificate (this needs to be created) or optionally a certificate signed by a trusted CA. This is a great way to setup the ultimate Jenkins CICD pipeline.

Latest Changes

Be sure to see the change log if interested in tracking changes leading to the current release. In addition, please refer to this article for even more details about this project.

Assumed Environment

It is assumed that the environment being used is Linux. The instructions within have been tested successfully on Ubuntu 16.10.

Getting Started

Important: It is imperative to follow these instructions in exact order as not doing so will result in problems and potential wasted time.

Ensure Docker Compose is installed along with Docker Engine. The included docker-compose.yml file uses version 3 so it's possible an upgrade of Docker Compose may be required.
Make sure the Java keytool is available for use. This can be accomplished by installing the openjdk-8-jre-headless package (e.g. sudo apt-get install openjdk-8-jre-headless).
Secure the Docker Daemon using TLS.
Create a Docker network named development.
Clone this repository into the desired location which will serve as the working directory moving forward.
In the Docker file for Jenkins Master, modify the -Duser.timezone setting found in the JAVA_OPTS environment variable to match the desired time zone. For more information, see this.
Generate a self-signed certificate using a private CA to use with the NGINX reverse proxy. If using a certificate from a trusted CA (non-private), still refer to the section on generating a self-signed certificate for instructions on where to place the files and other necessary changes.
Build the images by running the following command:

sudo make build
Prepare Jenkins Slave for successful certificate validation to use a different trust store so the certificate is successfully validated.

Note: This step can be skipped if using a certificate generated from a trusted CA (non-private). If you paid for a certificate from a trusted company, that certificate is likely already trusted by the default trust store.
Run the following command to create the Jenkins Master and NGINX containers:

sudo make run
Change the Jenkins URL to specify the address of the Jenkins installation which is accessible externally (e.g. https://jenkins.dev.internal.example.com:51205). This should match the FQDN of the certificate used to secure Jenkins via HTTPS. While logged into Jenkins, go to Manage Jenkins -> Configure System and then scroll down to the section labeled Jenkins Location. Enter the desired URL into the Jenkins URL field.

Note: If this step is missed, Jenkins will warn you of an invalid reverse proxy configuration.
Configure Jenkins to use ephemeral build slaves with JNLP.

Create a new pipeline job and enter the following for the script.

node ('testslave') {
  stage 'Stage 1'
  sh 'echo "Hello from your favorite test slave!"'
}

Save and run the job and take note of the results. If everything was setup properly, Docker should dynamically provision a Jenkins slave and then remove it when it's no longer needed.

Please read the rest of the content found within in order to understand additional configuration options.

Securing the Docker Daemon Using TLS

The following will configure the Docker Daemon using TLS. Before proceeding, be sure to research the official article from Docker on the subject.

Create a directory where the generated keys will be stored during the creation process.

cd ~

sudo mkdir docker

cd docker
Create the CA key by running the following commands.

sudo openssl genrsa -aes256 -out ca-key.pem 4096

The above command will require entering a passphrase to protect ca-key.pem. Enter a strong password and make note of what was entered as it'll be needed later. Then enter the next command:

sudo openssl req -new -x509 -days 365 -key ca-key.pem -sha256 -out ca.pem

The above command will request input in the following areas shown below.
```
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
```
It's important that for Common Name (e.g. server FQDN or YOUR name) to enter the hostname that the current OS is using (e.g. ubuntu-server). Also, keep in mind that the key will be valid for one year. If longer is desired, change the integer specified after -days.
Create the server key by running the following commands.

sudo openssl genrsa -out server-key.pem 4096

Then enter the next command:

sudo openssl req -subj "/CN=$HOST" -sha256 -new -key server-key.pem -out server.csr

For the above command, replace $HOST with the hostname that the current OS is using (e.g. ubuntu-server).

Then enter the next command:

echo subjectAltName = IP:$IPADDRESS,IP:127.0.0.1 > extfile.cnf

For the above command, replace $IPADDRESS with the IP address of the current host. Running the command ifconfig should show the current IP address.

Then enter the next command:

sudo openssl x509 -req -days 365 -sha256 -in server.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out server-cert.pem -extfile extfile.cnf

In the above command, keep in mind that the key will be valid for one year. If longer is desired, change the integer specified after -days.
Create the client key by running the following commands.

sudo openssl genrsa -out key.pem 4096

Then enter the next command:

sudo openssl req -subj '/CN=client' -new -key key.pem -out client.csr

Then enter the next command:

echo extendedKeyUsage = clientAuth > extfile.cnf

Then enter the next command:

sudo openssl x509 -req -days 365 -sha256 -in client.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out cert.pem -extfile extfile.cnf

In the above command, keep in mind that the key will be valid for one year. If longer is desired, change the integer specified after -days.
Run the following commands on all files ending with .pem (e.g. ca-key.pem, cert.pem, key.pem). This needs to happen for the reasons referenced in this discussion.

sudo mv ca-key.pem ca-key.bak

sudo mv cert.pem cert.bak

sudo mv key.pem key.bak

Then enter the next commands:

sudo openssl rsa -in ca-key.bak -text > ca-key.pem

sudo openssl rsa -in cert.bak -text > cert.pem

sudo openssl rsa -in key.bak -text > key.pem

Remove the .bak files as they are no longer needed.

sudo rm ca-key.bak cert.bak key.bak
Remove the certificate signing requests as they are no longer necessary.

sudo rm -v client.csr server.csr
Protect the keys and certificates by assigning the appropriate permissions.

sudo chmod -v 0400 ca-key.pem key.pem server-key.pem

Then enter the next command:

sudo chmod -v 0444 ca.pem server-cert.pem cert.pem
Copy the keys into the appropriate folder so Docker can use them.

sudo mkdir /etc/docker

Then enter the next command:

sudo cp ca.pem /etc/docker/.

Then enter the next command:

sudo cp server*.pem /etc/docker/.
Create and configure the docker.conf file by running the following commands.

sudo mkdir /etc/systemd/system/docker.service.d

Then enter the next command:

sudo vim /etc/systemd/system/docker.service.d/docker.conf

Next copy and paste the following into the file.
```
[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H unix:///var/run/docker.sock -D --tls=true --tlsverify --tlscacert=/etc/docker/ca.pem --tlscert=/etc/docker/server-cert.pem --tlskey=/etc/docker/server-key.pem -H tcp://0.0.0.0:2376
```
Reload and restart Docker by running the following commands.

sudo service docker stop

Then enter the next command:

sudo systemctl daemon-reload

Then enter the next command:

sudo service docker start

If everything worked correctly, issuing the command sudo docker ps should work fine without any errors.

Generating a Self-Signed Certificate Using a Private CA for NGINX

Following these instructions will create a private Certificate Authority with a server key. This key will be signed by the newly created Certificate Authority. In addition, Subject Alternate Names will be included so that one certificate can apply to multiple domain names and even IP addresses.

Note: If using a certificate generated from a trusted CA (e.g. non-private, you paid for a certificate from a trusted company), that certificate is likely already trusted by the default trust store. Therefore, this section may not apply (except for where to place the certificate files). However, it's important to ensure the certificate is configured in such a way that will work with the setup shown below. Please review this section carefully before requesting and paying for a certificate.

Make the necessary directories by running the following commands:

sudo mkdir ca

cd ca

sudo mkdir certs keys config

Important: Be sure to keep track of the absolute path of the CA folder. This will later be referred to as path_to_ca_files. Also, the absolute path used to clone this repository into will be referred to as path_to_repo_files. Be sure to replace both of these with the proper paths.
Copy server.cnf and ca.cnf into the config folder (e.g. /path_to_ca_files/config).
Edit server.cnf to include the appropriate domains under the alt_names section. Below is what is currently listed with an explanation.
```
[ alt_names ]
DNS.1                            = jenkins.dev.internal.example.com
DNS.2                            = *.internal.example.com
DNS.3                            = *.dev.internal.example.com
IP.1                             = 192.168.1.50
```
The DNS.1 entry is the primary FQDN that will be used externally (outside of the internal Docker network the containers use) to access Jenkins. Feel free to change this to your liking. However, be sure to update any wildcard entries to match. It is important to ensure the first entry is a FQDN and not in the form of a wildcard or problems may occur.

The additional DNS entries can each be a wildcard but keep in mind the asterisk will only apply to sub-domains at its current level. This is why DNS.2 and DNS.3 are formatted the way they are. For example, in the DNS.2 entry, the certificate will be valid for murmur.internal.example.com or gitlab.internal.example.com but not silly.dev.internal.example.com (in that case DNS.3 takes care of matching the sub-domain that silly is defined at).

Finally, change the IP.1 entry to match the IP address of the host running Docker. This will allow you to go to https://192.168.1.50:51205 without any warnings, assuming the certificate is trusted.

Note: Setting up a private DNS server may help with certain use cases. Please see this repository for help with that if applicable.
Based on the results from the previous step, modify docker-compose.yml to ensure each service name matches the domain naming scheme being used. For example, the name of each service by default is jenkins-master.dev.internal.example.com, jenkins-nginx.dev.internal.example.com, and jenkins-slave.dev.internal.example.com. The reason the service names use this format is it allows the applicable containers to securely communicate using HTTPS using a certificate that matches these names (in this case, the DNS.3 entry will match these).

Each service name works with the Docker Internal Name Resolution Service. In other words, inside the Docker internal network the containers run in, making a call to jenkins-nginx.dev.internal.example.com (this won't work externally) will resolve to the internal IP address that the Jenkins NGINX container uses. In order to prevent ignoring certificate validations, each service name must match any of the DNS entries defined in server.cnf.
Edit jenkins.conf and change line 21 to properly match the service name defined in docker-compose.yml for Jenkins Master.

Default Line 21 Entry: proxy_pass http://jenkins-master.dev.internal.example.com:8080;

Only change the FQDN and keep http:// and :8080; intact.
Edit nginx.conf and change line 20 to properly match the service name defined in docker-compose.yml for Jenkins Master.

Default Line 20 Entry: server jenkins-master.dev.internal.example.com:50000;

Only change the FQDN and keep :50000; intact.
Create the CA by running the following commands:

sudo openssl genrsa -aes256 -out /path_to_ca_files/keys/ca-key.pem 4096

You will be prompted to enter a password for ca-key.pem to protect it. Make sure to enter a good one and don't lose it.

sudo openssl req -new -x509 -config /path_to_ca_files/config/ca.cnf -days 365 -key /path_to_ca_files/keys/ca-key.pem -out /path_to_ca_files/certs/ca-cert.pem

Note: Feel free to change the integer after -days to match the length of time desired for the CA certificate to be valid.

You will be prompted to enter information such as Country Name, Organization Name, etc. Below are example entries.
```
countryName             = "US"
stateOrProvinceName     = "Texas"
localityName            = "Austin"
organizationName        = "Spacely Space Sprockets Inc."
organizationalUnitName  = "Spacely Space Sprockets CA"
commonName              = "Spacely Space Sprockets CA"
```
Create the server key and certificate signing request by running the following commands:

sudo openssl genrsa -out /path_to_ca_files/keys/server-key.pem 4096

sudo openssl req -subj "/CN=jenkins.dev.internal.example.com/O=server/" -sha256 -new -key /path_to_ca_files/keys/server-key.pem -out server.csr

Be sure the change jenkins.dev.internal.example.com to match the FQDN used to access Jenkins externally (this is different than the internal service names defined in docker-compose.yml) as discussed in the previous steps (e.g. DNS.1 entry in server.cnf).

Sign the public key with the CA and create the server certificate by running the following command:

sudo openssl x509 -req -days 365 -sha256 -in server.csr -CA /path_to_ca_files/certs/ca-cert.pem -CAkey /path_to_ca_files/keys/ca-key.pem \
    -CAcreateserial -out /path_to_ca_files/certs/server-cert.pem -extfile /path_to_ca_files/config/server.cnf -extensions v3_req

Note: Feel free to change the integer after -days to match the length of time desired for the server certificate to be valid.

Remove the certificate signing request by typing the following command:

sudo rm -v /path_to_ca_files/server.csr
Convert the CA certificate to DER encoded binary x.509 format by running the following command:

sudo openssl x509 -outform der -in /path_to_ca_files/certs/ca-cert.pem -out /path_to_ca_files/certs/ca-cert.crt
Protect the generated keys by making them readable only by you by typing the following command:

sudo chmod -v 0400 /path_to_ca_files/keys/ca-key.pem /path_to_ca_files/keys/server-key.pem
Protect the generated certificates by making them read only by typing the following command:

sudo chmod -v 0444 /path_to_ca_files/certs/ca-cert.pem /path_to_ca_files/certs/ca-cert.crt /path_to_ca_files/certs/server-cert.pem
Import the CA certificate into each applicable machine accessing Jenkins so as to prevent the certificate warning from being shown in the browser.

For Windows

Copy ca-cert.crt to each machine and then import it into the Trusted Root CA Certificates in certmgr. Go to run then type certmgr.msc then expand Trusted Root Certificatation Authorities. Right click on the folder named Certificates then select All Tasks -> Import. Find the ca-cert.crt copied to the machine and use that file for the import.

Restart the browser accessing the secured resource. The certificate warning message should no longer be present.

All Others

The steps to do this in other operating systems varies. If using Chrome, this article has many tips that will likely apply to most operating systems. It is suggested to do the necessary research based on your situation to accomplish this task since there is no one-sized fits all solution.
Copy the certificate files to the appropriate location by running the following commands:

sudo mkdir -p /path_to_repo_files/jenkins-nginx/volume_data/ssl

sudo cp /path_to_ca_files/certs/server-cert.pem /path_to_ca_files/keys/server-key.pem /path_to_repo_files/jenkins-nginx/volume_data/ssl

Preparing Jenkins Slave For Successful Certificate Validation

Important: Be sure to keep track of the absolute path of the CA folder. This will later be referred to as path_to_ca_files. Also, the absolute path used to clone this repository into will be referred to as path_to_repo_files. Be sure to replace both of these with the proper paths.

Create the directory to hold the new trust store (cacerts) by running the following command:

sudo mkdir -p /path_to_repo_files/jenkins-slave/volume_data/ssl
Copy the default cacerts file from Jenkins Slave to the newly created folder from the previous step by running the following commands:

sudo su

docker run --rm --entrypoint cat danieleagle/jenkins-slave:8u121-jdk-alpine /usr/lib/jvm/java-1.8-openjdk/jre/lib/security/cacerts > /path_to_repo_files/jenkins-slave/volume_data/ssl/cacerts

exit
Import the CA certificate created earlier (not the server certificate) into the new trust store by running the following command:

sudo keytool -noprompt -storepass changeit -keystore /path_to_repo_files/jenkins-slave/volume_data/ssl/cacerts -import -file /path_to_ca_files/certs/ca-cert.pem -alias MyPrivateCA

Feel free to give a more descriptive alias than MyPrivateCA.
Change the updated trust store to ready only to prevent accidental changes by running the following command:

sudo chmod -v 0444 /path_to_repo_files/jenkins-slave/volume_data/ssl/cacerts

Additional settings will be specified later when configuring the Yet Another Docker plugin to ensure the updated trust store is used by the Jenkins Slave.

Configuring Jenkins to Use Ephemeral Build Slaves

With the Docker Daemon secured using TLS and Jenkins Master running behind a NGINX reverse proxy using HTTPS, work can proceed to configure the Jenkins slave options.

Important: Be sure to keep track of the absolute path used to clone this repository. It will be referred to as path_to_repo_files. Be sure to replace this with the proper paths.

While in Jenkins (e.g. https://jenkins.dev.internal.example.com:51205), on the left sidebar go to Credentials and then on the sidebar below Credentials, click System.
Click on Global credentials (unrestricted) and then click Add Credentials in the left sidebar.
For Kind, select Docker Host Certificate Authentication and for the Scope, choose Global (Jenkins, nodes, items, all child items, etc).
Locate the client key used to secure the Docker Daemon. If it wasn't deleted, it should still be in /home/user/docker. Check by issuing the following commands.

cd ~/docker

ls

If key.pem is there then proceed to the next step. Otherwise, revisit Securing the Docker Daemon Using TLS.
Copy the contents of key.pem and paste into the Client Key field in Jenkins.
Locate the client certificate used to secure the Docker Daemon. It should be in /etc/docker/. Copy the contents of cert.pem and paste into the Client Certificate field in Jenkins.
Locate CA certificate used to secure the Docker Daemon. It should be in /etc/docker/. Copy the contents of ca.pem and paste into the Server CA Certificate field in Jenkins.
Click OK when finished.
On the left sidebar, click Manage Jenkins then Configure System.
Scroll down to the section titled Cloud and find Yet Another Docker. For the Cloud Name field, enter a desired name (e.g. hostname of Docker Machine) and for the Docker URL field, enter tcp://192.168.1.50:2376. Be sure to replace the IP address with that of the one running Docker Machine.
Under the Host Credentials field, select the recently added credentials created in the previous steps.
Under the Type field, select NETTY and then click on the Test Connection button. If no errors are displayed, move on to the next step. Otherwise, retrace/retry previous steps.
Under the Max Containers field, the default is 50. This is the maximum amount of Jenkins slave containers that will be provisioned at any given time. Change this value to the desired amount or leave it as default.
Under the Images section, click the Add Docker Template button and select Docker Template. For the Docker Image Name, enter danieleagle/jenkins-slave:8u121-jdk-alpine.
Under the Pull Image Settings section, locate the Pull Strategy field and select Pull never. Since the image is local there is no need to pull it, so ensure this setting is set correctly.
Under the Create Container Settings section, click the Create Container settings... button. Scroll down to Volumes and enter /path_to_repo_files/jenkins-slave/volume_data/ssl:/etc/ssl/java/truststore:ro into the field. Change path_to_repo_files to the folder where you cloned this repository. It's important to make sure to use the absolute path instead of relative.

Note: This step can be skipped if using a certificate generated from a trusted CA (non-private). If you paid for a certificate from a trusted company, that certificate is likely already trusted by the default trust store. In that case, referring to a different trust store is likely unnecessary.
While still in the Create Container Settings section, scroll down to Network Mode and enter development into the field.
Under the Remove Container Settings section, check Remove volumes.
For the Labels field, enter testslave.

Note: This will likely be a different name later on when moving past the testing phase of this initial setup. This name should match the node found in the pipeline script.
Under the Usage field, select Only build jobs with label expressions matching this node.
Under the Launch method field, select Docker JNLP launcher.
Under the Linux user field, enter jenkins.
Under the Slave JVM options field, enter -Xmx8192m -Djava.awt.headless=true -Duser.timezone=America/Chicago -Djavax.net.ssl.trustStore=/etc/ssl/java/truststore/cacerts. Be sure the change the timezone to the appropriate value.

Note: If using a certificate generated from a trusted CA (non-private), the string -Djavax.net.ssl.trustStore=/etc/ssl/java/truststore/cacerts can be omitted. If you paid for a certificate from a trusted company, that certificate is likely already trusted by the default trust store. In that case, referring to a different trust store is likely unnecessary.
Under the Different jenkins master URL field, enter https://jenkins-nginx.dev.internal.example.com. Be sure to change the format of this URL based on the applicable domain being used (defined as the Jenkins NGINX service name in docker-compose.yml). This was discussed earlier when creating a self-signed certificate using a private CA.
When done with everything, click the Save button at the bottom of the page.

Jenkins Secure Email with TLS

When setting up email using TLS for notifications, alerts, etc., be sure to uncheck SSL but instead use port 587. This will still send email securely using TLS and get around any resulting errors from checking the SSL box but using it with TLS. The Dockerfile for Jenkins Master adds -Dmail.smtp.starttls.enable=true to the JAVA_OPTS environment variable to ensure TLS will work.

Forwarding JNLP Traffic

Since Jenkins Master is configured to use a NGINX reverse proxy with HTTPS, all HTTP traffic directed at Jenkins will go through this proxy so that HTTPS is enforced. In addition, a special configuration setting had to be specified for NGINX to forward the appropriate JNLP traffic for use by Jenkins slaves. This was configured by adding the following to the nginx.conf file.

stream {
  server {
    listen 50000;
    proxy_pass jenkins;
  }

  upstream jenkins {
    server jenkins-master.dev.internal.example.com:50000;
  }
}

Container Network

The network specified (can be changed to the desired value) by these Docker containers is named development. It is assumed that this network has already been created prior to using the included Docker Compose file. The reason for this is to avoid generating a default network so that other Docker containers can access the services these containers expose using the Docker embedded DNS server.

If no network has been created, run the following Docker command: sudo docker network create network-name. Be sure to replace network-name with the name of the desired network. For more information on this command, go here.

Port Mapping

The external ports used to map to the internal ports that Jenkins uses are 51205 (maps to 443 for HTTPS) and 51206 (maps to 50000 for JNLP). These ports can certainly be changed but please be mindful of the effects. Additional configuration may be required as a result.

Data Volumes

It is possible to change the data volume folders mapped to the Jenkins Master container to something other than ./volume_data/x if desired. It is recommended to choose a naming scheme that is easy to recognize. Additional configuration may be required as a result.

Notes About the Included Make File

Instead of accessing Docker Compose directly, a makefile is included and should be used instead. The reason for this is that the Jenkins slave shouldn't be running right away; it should only run when it's required. All that will be handled dynamically by the Yet Another Docker Plugin which is included with the Jenkins Master container. Thus, using the makefile will prevent the Jenkins slave from running right away.

Logging

In order to properly rotate the logs that Jenkins outputs, logrotate can be used. Included is an example logrotate file for Jenkins Master as well as a file for Jenkins NGINX that can be copied to /etc/logrotate.d so that the logs get rotated based on the settings specified in those files.

Also, another approach would be to create a file with logrotate settings for all Docker containers and copy it to /etc/logrotate.d. This would rotate all the logs for all Docker containers.

/var/lib/docker/containers/*/*.log {
  rotate 52
  weekly
  compress
  size=1M
  missingok
  delaycompress
  copytruncate
}

Finally, if running in a production environment where logs are closely monitored, it's recommended to use something like Fluentd. This aggregates all your logs and makes them easily searchable. Granted, Fluentd does much more than this so it's recommended to check out the official docs. The great news is Docker has a native Fluentd logging driver.

If Fluentd has been setup and you wish to use it, docker-compose.yml will need to be modified to ensure it uses it instead of the default logging driver. Under each defined service, add the following:

logging:
  driver: fluentd
    options:
      fluentd-address: localhost:24224
      tag: "{{.ImageName}}/{{.Name}}/{{.ID}}"

Jenkins Plugins

The file plugins.txt is used to install plugins for Jenkins when creating the Jenkins Master container. Additional plugins can be added (they can be removed as well) to this file before creating the container. Also, from time to time the plugins listed within this file may become out of date and require specifying the newest versions.

Jenkins Slave OS

It is possible to use a different OS for Jenkins slaves. However, it will require a specific configuration that works for the desired OS. In addition, the OS should work with the plugin used for the Jenkins slaves, Yet Another Docker Plugin. Thus, it will need to make use of OpenJDK.

Jenkins Slave Tools

The included Jenkins slave container doesn't have all the tools needed for every given objective or build task. It will need to be modified to include the appropriate tools. In addition, certain Jenkins plugins may need to be installed in order to achieve a specific task to complement the Jenkins slave.

Using This Solution with Docker Swarm

It is possible to adapt this solution for use with Docker Swarm. Take a look at this article for details.

Special Thanks

Special thanks goes to David Hale for streamlining the process to secure the Docker Daemon and Maxfield Stewart as mentioned above for his amazing article that served as inspiration for this solution.

karthik101/jenkins-ephemeral-cicd