/docker-devops

Create and configure infrastructure (Elastic Stack and CI/CD platforms) in AWS by executing Terraform and Ansible playbooks in Docker containers via Jenkins 2 declarative pipeline

Primary LanguageHCL

This is Work In Progress

Objectives

  • Create and configure a DevOps environment in AWS
  • Use Open Source Software (OSS) or free trial versions as much as possible
  • The DevOps environment created will need to be reproducible and predictable
  • The DevOps environment created will consist of two Swarm clusters (VPC peering) to support the following:

In order to achieve the stated objectives above, the following will need to be automated via a Jenkins/2 declarative pipeline. The sole purpose of this stand-alone Jenkins instance is to recreate the DevOps environment. This Jenkins instance can be deployed anywhere that has docker and internet access:

  • Recreate (terminate / create) the infrastructure using Terraform
  • Configure the infrastructure using Ansible
  • Deploy the application stacks (CI/CD + Elastic Stack) in docker swarm mode

Pipeline picture to be added

NOTE:

The Jenkins/2 pipeline to achieve the above objectives have been tested in a Vagrant CentOS/7 VM on a Windows 10 pro machine only

Prerequisites

  1. See Prerequisites in the ./terraform directory
  2. Fork this repository in GitHub as your forked repo will need to be integrated with Jenkins. Otherwise, you will not be able to modify the pipeline as per you own needs
  3. The items below are only required if you are using a Windows 10 machine that does not have Docker for Windows installed on it:
    1. Windows 10 pro machine to act as a dev environment from where to launch the automated infrastructure pipeline
    2. latest versions of VirtualBox, Vagrant and Git BASH are installed on the dev machine
    3. Install vagrant-vbguest plugin: vagrant plugin install vagrant-vbguest
    4. Install vagrant-hostmanager plugin: vagrant plugin install vagrant-hostmanager

Create a CentOS/7 VM on a Windows machine

It is assumed you have a stable internet connection:

  1. Start a Git Bash shell and create a suitable directory and then change to it:
    • mkdir -p /tmp/github/docker-devops
    • cd /tmp/github/docker-devops
  2. Checkout the Vagrantfile: curl -o Vagrantfile https://raw.githubusercontent.com/shazChaudhry/docker-devops/master/Vagrantfile
  3. Create a CentOS/7 VM in a VirtualBox and then SSH to it: clear; vagrant destroy --force; vagrant box update; vagrant box prune; vagrant up; vagrant ssh
  4. You are now ready to deploy a stand-alone Jenkins instance in this VM

Deploy a stand-alone Jenkins instance

It is assumed that Jenkins is being deployed in a secure environment that has internet access and docker installed and that you have followed the prerequisites in ./terraform/README.md

  1. Start jenkinsci/blueocean by following the command below in your terminal:
        docker container run -d \
        --rm \
        --user root \
        --name jenkins \
        --publish 8080:8080 \
        --volume jenkins-data:/var/jenkins_home \
        --volume /var/run/docker.sock:/var/run/docker.sock \
        --volume $HOME:/root \
        jenkinsci/blueocean
  1. After a few moments, Jenkins should be accessible at http://IP_ADDRESS:8080 or at http://devops:8080 if it is running inside a VM that you created above
  2. You will need an adminstrator password in order to unlock Jenkins: docker container exec -it jenkins cat /var/jenkins_home/secrets/initialAdminPassword
  3. Once Jenkins has been unlocked with the admin password, select "Install suggested plugins" on customize Jenkins page. You will then need to follow the on-screen instructions to complete the setup

Integrate Jeknins with the GitHub repo that you forked as part of the prerequisites above

See "Create your Pipeline project in Blue Ocean" section at https://jenkins.io/doc/tutorials/create-a-pipeline-in-blue-ocean/ for instructions on te integration

Improvements

In this repository, I have used Terraform and Ansible in the infrastructure provisioning and configuring part of the solution as these skills seem to be a lot in demand at the moment. However, for the purpose of achieving the stated objectives above, Terraform and Ansible are strictly speaking not required. Adding layers of these technologies has made the entire solution unnecessarily more complex and difficult to manage.

Instead of using Terraform and Ansible, a production grade self-healing infrastructure could much easily be created using Docker for AWS. In my honest opinion, "Docker for AWS" is a much better option in this scenario as creating a Swarm cluster is dead simple.

Please visit "Docker for AWS setup & prerequisites" page for instructions on how to create a Swarm cluster in AWS or Azure