DevOps for Blockchain using Jenkins
This project is inspired by the work of Redback and Microsoft teams for developing the process using VSTS.
I've chosen to develop a similar DevOps process with Jenkins, Truffle and TestRPC so that we can give choices for our customers.
The project contains the components needed to create the test pipeline.
Building Jenkins Server
The current version is based on a single instance of Jenkins running Ubuntu 17.04. The scripts can be adapted for other Linux distributions as needed.
The script to automate the deployment is located at jenkins/master-install.sh. Jenkins installation is split into 2 routines. First, install the dev tools needed to compile and deploy Java and Node applications such as JDK, Node, npm, Git and Jenkins itself.
Followed by tools needed for running Ethereum locally on the Jenkins server. We'll be using Truffle Framework for writing contracts, integration tests and running tests. testrpc will provide an in-memory blockchain environment that's fast and predictable.
I've chosen to write the tests in Mocha as it's a versatile framework that can be used not just for Smart Contracts, but for integration testing across the entire application stack, thus allowing the developers to learn less and do more. The results are written into XML using the Mocha JUnit Reporter. The XML results can then be pulled into Jenkins as part of the build process.
Install Dev Tools
apt-get install -y openjdk-8-jdk-headless
apt-get install -y git
wget -q -O - https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo apt-key add -
echo deb http://pkg.jenkins.io/debian-stable binary/ | sudo tee /etc/apt/sources.list.d/jenkins.list
apt-get update
apt-get install -y jenkins
apt-get install -y python-software-properties
curl -sL https://deb.nodesource.com/setup_7.x | sudo -E bash -
apt-get install -y nodejs
apt-get install -y npm
Install Ethereum Tools
npm install -g ethereumjs-testrpc
npm install -g solc
npm install -g truffle
npm install -g mocha
npm install -g mocha-junit-reporter
mkdir -p /opt/ethereum/web3j
wget https://github.com/web3j/web3j/releases/download/v2.3.1/web3j-2.3.1.tar -O /tmp/web3j.tar
tar -xf /tmp/web3j.tar --directory /opt/ethereum/web3j --strip-components=1
Install Jenkins and Tools using Azure Resource Manager (ARM) Template
The full VM deployment and configuration can be automated through an ARM template. The template is located in jenkins/arm/deploy-ubuntu-master.json. This template will:
- Setup a new virtual network, subnet, network security groups
- Deploy an Azure Marketplace Ubuntu 17.04 VM with public IP address
- Install Jenkins, Dev Tools and Ethereum Tools - this script is located in jenkins/install-jenkins-master-ubuntu.sh
You can customize the ARM template such that it can be:
- Deployed into an existing environment
- Restrict access through network security groups
- Replace username/password for the VM with SSH Keys
To deploy using Azure Portal
- Login to Azure Portal
- From the left ribbon, search for Templates
- Create a new template
- Copy & Paste the contents of the JSON file
- Save
- Click Deploy, follow the prompts to completion
To deploy through Azure CLI
# Start CLI in interactive mode & login
az interactive --style none
az login
# Create a Resource Group
az group create -l eastus -n blockchain-jenkins
# Deploy ARM template & enter the VM Username & Password
az group deployment create --resource-group blockchain-jenkins --template-uri https://raw.githubusercontent.com/SenthuranSivananthan/devops-blockchain-jenkins/master/jenkins/arm/deploy-ubuntu-master.json
Wait about 10 minutes for the deployment process to complete.
# Retrieve the Public IP address of the Jenkins Server
az network public-ip show --resource-group blockchain-jenkins --name pip-jenkins-master
Access Jenkins via http://[public ip]:8080/
To deploy through PowerShell
Coming soon
Using Truffle Framework
An example Smart Contract and it's corresponding integration tests are located in the (/ethereum)[/ethereum].
To initialize a new project
cd <project directory>
truffle init
You should now have a few directories & files:
- contracts/ - stores all Smart Contracts
- tests/ - stores all test cases
- migration/ - contracts that allow for deploying Smart Contracts to Blockchain
- truffle.js - truffle configuration file
The Smart Contracts are written in Solidity and will store the files with .sol extension. There are example test cases written in Solidity (.sol extension) and Mocha (.js extension). The choice of the language is up to you.
Executing Integration Tests
You can run the integration tests against your own new project or the example Smart Contract(s) located in this project.
cd <project directory>
# Start testrpc in the background and save the pid. You can set this up on another terminal as well.
testrpc &
echo $! >> testrpc.pid
# Run the tests
truffle test
# Stop testrpc
kill -9 `cat testrpc.pid`
Exporting Integration Test Results
You can export the integration tests into multiple formats. For this project, I'm using JUnit Reporter so that it can be read by Jenkins. The configuration is located in truffle-jenkins.js. To use this version, rename truffle-jenkins.js to truffle.js.
Test Automation
I've chosen to use Jenkinsfile to declaratively configure the build pipeline. This allows for the build definitions to be version controlled along with the source code.
The pipeline is split into 3 stages:
- Launching testrpc
- Executing Smart Contract tests
- Publishing test results using JUnit
In the future, we can add additional stages to integrate with the full application stack for end to end testing. The stages allow for easier monitoring & troubleshooting as builds are executed:
Since we are using Mocha JUnit Reporter with Truffle, Jenkins is able to process the test-results.xml
and display the test status as part of the dashboard.
And you will be able to drill down into each test case for deeper look.
When Smart Contract tests fail, the error information is displayed directly in Jenkins, thus aligning to existing developer testing/workflows:
More to come ...