odennav/server-health-monitoring-observability

Server Health Monitoring and Observability

The Objective of this project is to simulate automated monitoring and observability of Cellusys servers by retrieving information on system resource utilizations(cpu, memory, disk, power) and status of services(elk, ntp).

The Linux servers are classified as:

Central servers
Message processors

Ansible is used to setup deployment of system-resource monitoring scripts in all central servers and message processors.

Secure continuous integration and continuous delivery pipeline configured in Jenkins. System resource-usage and service status messages are sent to a Slack channel for real-time observability of resources and services.

Getting Started

Supported systems:

RedHat
CentOS

We'll implement workflow below:

Provision Servers
User Configuration on Linux servers
Docker Installation
Slack Setup
Jenkins Installation and Configuration
SonarQube Installation and Setup
Gogs Installation and Configuration
Trivy Installation and Integration
Ansible Setup and Deployment
Install Plugins
Pipeline Setup with Jenkinsfile

Provision Servers

Eight linux servers are provisioned with Vagrant in this lab. Use Vagrantfile in this repository.

Install Vagrant

If you haven't installed Vagrant, download it here and follow the installation instructions for your OS.

If you encounter an issue with Windows, you might get a blue screen upon attempt to bring up a VirtualBox VM with Hyper-V enabled.

To use VirtualBox on Windows, ensure Hyper-V is not enabled. Then turn off the feature with the following Powershell commands:

Disable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-All
bcdedit /set hypervisorlaunchtype off

After reboot of your local machine, run:

vagrant up

User Configuration on Linux Servers

Add New User

We'll use cs1 virtual machine as our build machine. Integrations to pipeline is implemented on this server

Change password for root user

Login to Vagrant VM

vagrant ssh cs1

sudo passwd

Switch to root user. Add new user 'odennav' to sudo group.

sudo useradd odennav
sudo usermod -aG wheel odennav

Notice the prompt to enter your user password. To disable password prompt for every sudo command, implement the following:

Add sudoers file for odennav-admin

echo "odennav ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/odennav

Ensure correct permissions for sudoers file

sudo chmod 0440 /etc/sudoers.d/odennav
sudo chown root:root /etc/sudoers.d/odennav

Test sudo privileges by switching to new user

su - odennav
sudo ls -la /root

To change the PermitRootLogin setting, modify the SSH server configuration file /etc/ssh/sshd_config as shown below:

PermitRootLogin no

Please note you'll have to repeat this user setup for each server provisioned.

Docker Installation

Uninstall any older versions before installing a new version, along with associated dependencies

sudo yum remove docker \
                  docker-client \
                  docker-client-latest \
                  docker-common \
                  docker-latest \
                  docker-latest-logrotate \
                  docker-logrotate \
                  docker-engine

Install the yum-utils package and set up the repository

sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

Install Docker Engine, containerd, and Docker Compose

sudo yum install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Start Docker after installation

sudo systemctl start docker

Verify successful Docker engine installation

sudo docker run hello-world

Slack Setup

Slack is the communication platform on our local machine we'll use to receive resource usage notifications from monitored servers.

Procedure

Download Slack here for Windows, Mac or Linux
Create new workspace
Setup new group channel in your workspace
Enable and create incoming webhooks to your group channel. Use this guide for further reference
Select the channel your slack app will post to and a Webhook URL will be generated as shown.

This URL is used in our monitoring script for HTTP POST requests.

Jenkins Installation and Configuration

Install Jenkins

We'll use the long term support release which is installed from redhat-stable yum repository.

sudo wget -O /etc/yum.repos.d/jenkins.repo \
    https://pkg.jenkins.io/redhat-stable/jenkins.repo
sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io-2023.key
sudo yum upgrade
# Add required dependencies for the jenkins package
sudo yum install fontconfig java-17-openjdk
sudo yum install jenkins
sudo systemctl daemon-reload

Enable jenkins service

sudo systemctl enable jenkins

Start jenkins service

sudo systemctl start jenkins

Confirm jenkins service is active and running

sudo systemctl status jenkins

Post Installation Setup

Next we use the post-installation setup wizard to unlock jenkins, customize plugins and create first admin user required to continue accessing jenkins.

Browse to 192.168.10.6:8080 to see the Unlock Jenkins page.
Obtain the automatically-generated alphanumeric password

sudo cat /var/jenkins_home/secrets/initialAdminPassword

Paste this password into the Administrator password field and click Continue to access jenkin's main UI.

Customize Jenkins with Plugins

After unlocking Jenkins, the Customize Jenkins page appears.

Here you can install any number of useful plugins as part of your initial setup.

Click on Install suggested plugins to install the recommended set of plugins, which are based on most common use cases.

Create First Administrator User

Finally, after customizing Jenkins with plugins, Jenkins asks you to create your first administrator user.

When the Create First Admin User page appears, specify the details for your administrator user in the respective fields and click Save and Continue.
When the Jenkins is ready page appears, click Start using Jenkins. Notes:

This page may indicate Jenkins is almost ready! instead and if so, click Restart.

If the page does not automatically refresh after a minute, use your web browser to refresh the page manually.

If required, log in to Jenkins with the credentials of the user you just created and you are ready to start using Jenkins.

SonarQube Installation and Setup

SonarQube is a code quality assurance tool that collects and analyzes source code, providing reports for the code quality of our project.

It enables us to deploy clean code consistently and reliably.

We'll run the long term community version of sonarqube's image.

Install SonarQube Container

docker run -d --name sonar -p 9000:9000 sonarqube:lts-community

Confirm container is running

docker ps -a --filter "name=sonar"

Browse to UI of SonarQube at 192.168.10.1:9000

Use admin for default username and password. Update to new password when requested.

Generate Token

Go to Administration tab and select Security tab.

From the drop down, click on Users.

This section is used to create and administer individual users.

Click on button at far right under Tokens column.

Enter Token Name as sonar-token and click on Generate. Note period of expiry.

Copy this access code, we'll use it to create credential for SonarQube in jenkins.

Create SonarQube Secret

Go to Jenkins dashboard and select Manage Jenkins.

Under the Security section, select Credentials

Click on (global) domain of jenkins System store

Next, click on blue button + Add Credentials at top right

Assign the following:

Kind -------------------------> Secret text

Scope ------------------------> Global

Secret -----------------------> sonar-token

Description ------------------> sonar

Configure SonarQube Server

Go to Jenkins dashboard and select Manage Jenkins.

Under the System Configuration section, select Configure System

Scroll down and search for SonarQube servers and SonarQube installatons

Click Add SonarQube and assign the following:

Name ---------------------------------> sonar-server

Server URL ---------------------------> 192.168.10.1:9000/

Server authentication token ----------> sonar

Click on Save

Gogs Installation and Configuration

Build a simple, stable and extensible self-hosted Git service.

Pull image from Docker Hub.

docker pull gogs/gogs

Create local directory for volume.

mkdir -p /opt/gogs

Use docker run for the first time.

docker run --name gogs --restart always -p 10022:22 -p 3880:3000 -v /opt/gogs:/data gogs/gogs

It is important to map the SSH service from the container to the host and set the appropriate SSH Port and URI settings when setting up Gogs for the first time.

To access and clone Git repositories with the above configuration you would use:

git clone ssh://git@192.168.10.101:10022/odennav/server-health-monitoring.git

Files will be store in local path of build-machine instance, /opt/gogs in my case.

For first-time run installation, install gogs with mysqllite3

Initialize local repository and create README

git init
touch README.md
echo "Server Health Monitoring" > README.md

git config --global user.email "odennav@odennav.com"
git config --global user.name "odennav-gogs"
git config --global credentials.helper store

Add all changes to staging area and commit

git add .
git commit -m "first commit"

Connect local repo with remote repository

git remote add origin https://192.168.10.101:3880/odennav/server-health-monitoring.git

Push commits to remote repository

git push -u origin master

Set tracking information for this branch

git branch --set-upstream-to=origin/master master

Add Public SSH Key to Gogs Server

We'll add RSA public key to Gogs to ensure SSH authentication with Jenkins.

Generate RSA key pair

ssh-keygen

Select the defaults for all three prompts by hitting the Enter key at each prompt.

By default, ssh-keygen will save the key pair to ~/.ssh directory

In that directory, two files id_rsa and id_rsa.pub corresponding to the private and public keys, respectively will be present.

Go to Gogs settings page at http://192.168.10.1:3880/user/settings

Select SSH Keys tab and click on Add Key on Manage SSH Keys tab
Enter Key Name as id_rsa

Copy your public key, paste into Content field and click Add Key

cat ~/.ssh/id_rsa.pub | tr -d '\n'

Add Private Key as Gogs Credential to Jenkins

Here we'll add private RSA key generated on cs1 where jenkins-master is installed and add to Jenkins.

Go to Jenkins dashboard and select Manage Jenkins.

Under this section, select Credentials

Click on (global) domain of jenkins System store

Next, click on blue button + Add Credentials at top right

Assign the following:

Kind -------------------------------> SSH Username with private key

Scope ------------------------------> Global

Description ------------------------> private-key

Username ---------------------------> odennav-gogs

Select radio button Enter directly for Private Key

Click on Add button on the right and paste the copied private key as new secret into key field.

Select Create to save credential.

Trivy Installation and Integration

Trivy is an open source security scanner used to find vulnerabilities and Iac misconfigurations.

It can be used to scan the following:

Container images
Filesystem
Virtual machine image
Git repository
Kubernetes cluster
Cloud infrastructure

Install using package manager

cat << EOF | sudo tee -a /etc/yum.repos.d/trivy.repo
[trivy]
name=Trivy repository
baseurl=https://aquasecurity.github.io/trivy-repo/rpm/releases/\$basearch/
gpgcheck=1
enabled=1
gpgkey=https://aquasecurity.github.io/trivy-repo/rpm/public.key
EOF
sudo yum -y update
sudo yum -y install trivy

If you intend to use trivy as container image, mount docker.sock as from the host into the Trivy container.

Integrate Trivy to Jenkins with Bash Script

The bash script below is used in our groovy pipeline script to scan and analyze gogs repositrory with Trivy for vulnerabilities.

#!/bin/bash


# Environment variables
TEMPLATE_PATH="@/home/odennav/opt/gogs/trivy/html.tpl"
DATE=$(date +"%Y-%m-%d_%H-%M")


echo "Starting Trivy Scan"

vulnScan() {

# Trivy scan command on filesystem for gogs repository on cs1
    trivy fs --security-checks vuln,secret,misconfig /opt/gogs/ --format template --template ${TEMPLATE_PATH} --output trivy_report_$DATE.html

    exit_code=$?
    echo "Exit code from Trivy scan: $exit_code"

    if [ "$exit_code" -eq 1 ]; then
        echo "Vulnerability scan of server monitoring scripts failed"
        exit 1;
    else
        echo "No vulnerabilities found"
    fi
}

#Main script

vulnScan

When jenkins pipelne is executed, if there are any vulnerabilities, the next steps in deploy pipeline will be stopped from running.

Ansible Setup and Deployment

Install Ansible

To install ansibe without upgrading current python version, we'll make use of the yum packae manager

sudo yum update

Install EPEL repository

sudo yum install epel-release

Verify installation of EPEL repository

sudo yum repolist

Install Ansible

sudo yum install ansible

Confirm installation

ansible --version

Another approach to install Ansible, we'll be to use bash script to upgrade current python version. Check this repository for python script python_upgrade.sh

Upgrade python

./python_upgrade.sh

Confirm new python version

python3 -V

Install Ansible with pip

python3 -m pip install --user ansible

Install ansible-core package

$ python3 -m pip install --user ansible-core

Confirm installation

ansible --version

Configure Ansible Vault

Ansible communicates with target remote servers using SSH and usually we generate RSA key pair and copy the public key to each remote server, instead we'll use username and password credentials of odennav user.

This credentials are added to inventory host file but encrypted with ansible-vault

Ensure all IPv4 addresses and user variables of remote servers are in the inventory file as shown

View ansible-vault/values.yml which has the secret password

cat /server-health-monitoring/ansible-vault/values.yml

Generate vault password file

openssl rand -base64 2048 > /server-health-monitoring/ansible-vault/secret-vault.pass

Create ansible vault with vault password file

ansible-vault create /server-health-monitoring/ansible-vault/values.yml --vault-password-file=/server-health-monitoring/ansible-vault/secret-vault.pass

View content of ansible vault

ansible-vault view /server-health-monitoring/ansible-vault/values.yml --vault-password-file=/server-health-monitoring/ansible-vault/secret-vault.pass

Read ansible vault password from environment variable

export ANSIBLE_VAULT_PASSWORD_FILE=/server-health-monitoring/ansible-vault/secret-vault.pass

Confirm environment variable has been exported

export ANSIBLE_VAULT_PASSWORD_FILE

Test Ansible by pinging all remote servers in inventory list

ansible all -m ping

Run ansible playbook

Playbook deploy_bundle/deploy_bundle.yml will implement the following tasks in remote servers:

Create directories for each monitoring role
Copy shell scripts to remote hosts for monitoring
Setup cron jobs to execute scripts in remote hosts
Restart cron service

Install Plugins

Plugins are required to integrate tools to Jenkins and execute in our pipeline script.

Go to Plugin Manager under Manage Jenkins section of Jenkins dasboard

Our next task is to search and install the following plugins below:

Eclipse Temurin installer
Docker
Docker Pipeline
docker-build-step
CloudBees Docker Build and Publish
Gogs
Trivy
Ansible

Select Install without restart at bottom left

Configure Other Global Tools

Go to Global Tool Configuration under Manage Jenkins section of Jenkins dasboard

Note procedures to configure jdk and docker as global tools.

Procedure - JDK Scroll down and search for JDK and JDK installations

Click on Add JDK

Enter or select the following:

Name -----------------------------> jdk11
Install automatically ------------> ✔️
Add Installer --------------------> Install from adoptium.net
Version --------------------------> jdk-11.0.19+7

Procedure - Docker Scroll down and search for Docker and Docker installations

Click on Add Docker

Enter or select the following:

Name -----------------------------> docker
Install automatically ------------> ✔️
Add Installer --------------------> Download from docker.com
Docker version --------------------------> latest

Click on Apply to save configuration.

Note, we'll need to also configure global tools for Trivy and Ansible.

Pipeline Setup with Jenkinsfile

Jenkins Pipeline is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins.

This continuous delivery (CD) pipeline is an automated expression of our process for getting the monitoring scripts from Gogs right through to our remote hosts.

Setup Pipeline

Implement procedure below:

Go to Jenkins main dashboard and click on New Item
Name pipeline, select Pipeline as type of project and click OK
Click on the created job and scroll down to the “Pipeline” section in the configuration screen.
Choose “Pipeline script from SCM” and select type of SCM.
Enter the URL of the gogs repository containing your Jenkinsfile.
Add credentials of gogs repository.
Choose the branch to build from, typically /main or /master
Specify the path of Jenkinsfile in SCM as /server-health-monitoring/Jenkinsfile
Click on Save to save this configuration.
Restart Jenkins to apply configuration changes or updates effectively.
Navigate to the Jenkins “dashboard” and click on Manage Jenkins in the sidebar.
Select Reload Configuration from Disk or Restart Safely.

The pipeline block below defines all the work done throughout our entire Pipeline.


pipeline {
    agent any
    tools{
        jdk 'jdk11'
    }
    environment{
        SCANNER_HOME= tool 'sonar-scanner'
        HEALTH_CHECK_SCRIPTS_PATH="/opt/gogs/server-health-monitoring/health_check_scripts"
        TRIVY_SCRIPT_PATH="/opt/gogs/server-health-monitoring/trivy"
        ANSIBLE_DEPLOY_SCRIPT_PATH="/opt/gogs/server-health-monitoring/deploy_bundle/deploy_bundle.yml"    
        ANSIBLE_VAULT_PATH="\"@/server-health-monitoring/ansible-vault/values.yml\""
}
    stages {
        stage('Git Checkout') {
            steps {
                git branch: 'master', credentialsId: 'odennav-gogs', url: 'https://192.168.10.1:3880/odennav/server-health-monitoring-observability.git'
            }
        }
        stage('Slack Webhook Integration') {
            steps {
                withCredentials([string(credentialsId: 'slack_webhook_url', variable: 'URL')])
                sh "sed -i 's|webhook_url|${URL}|g' $HEALTH_CHECK_SCRIPTS_PATH/*.sh   
            }
        }
        stage('SonarQube Scan') {
            steps {
                withSonarQubeEnv('sonar-server'){
                    sh '''$SCANNER_HOME/bin/sonar-scanner -Dsonar.projectName=Server-Health-Monitoring \ 
                    -Dsonar.java.binaries=. \
                    -Dsonar.projectKey=Server-Health-Monitoring'''
                }    
            }
        }
        stage('Trivy Vulnerability Scan') {
            steps {
                sh "sudo bash $TRIVY_SCRIPT_PATH/trivy_repo_scan.sh"    
            }
        }
        stage('Ansible Deployment') {
            steps {
                sh "ansible-playbook --inventory inventory  $ANSIBLE_DEPLOY_SCRIPT_PATH -e $ANSIBLE_VAULT_PATH"     
            }
        }
    }
    post {
        always {
            archiveArtifacts artifacts: "trivy_report_*.html", fingerprint: true    
            publishHTML (target: [
                allowMissing: false,
                alwaysLinkToLastBuild: false,
                keepAll: true,
                reportDir: '.',
                reportFiles: 'trivy_report_*.html',
                reportName: 'Trivy Scan',
                ])
            }
        }
}

When all scans are passed, the Ansible Deployment stage will proceed to setup cron jobs for health monitoring scripts in remote hosts.

Build Pipeline

Select pipeline created above and trigger a build of pipeline job.

Click on Build Now
Jenkins will fetch the Jenkinsfile from gogs repository and run jobs as defined.
View the progress of the pipeline job on the Jenkins dashboard.
Click on the job to view detailed logs and status updates as each stage of the pipeline is executed.

Check the console output and logs for more info on any failures.

SAST Reports

To view reports generated by SonarQube:

Browse to 192.168.10.1:9000 and click on Projects tab.
Select project we created in pipeline script Server-Monitoring
View bugs, vulnerabilities, code smells, duplications and hotspots reviews.

To view scan reports from Trivy:

Select pipeline job created
Scroll down and click on Trivy Scan tab on the left bar
View vulnerabilities found with different severity levels.

Jenkins Slave Setup(Optional)

We'll use node named jenkins-agent in Vagrantfile to run as slave node to Jenkins.

Login to js node with 192.168.10.9 assigned as its IPv4 address in Vagrantfile.

vagrant up js
vagrant ssh js

Switch to root user

su -

Install Java

sudo yum install fontconfig java-17-openjdk

Make root working directory for Jenkins

cd ~
sudo mkdir jenkins-slave

Change permissions of directory

sudo chmod 755 ~/jenkins-slave

Generate RSA key pair

ssh-keygen

Select the defaults for all three prompts by hitting the Enter key at each prompt.

Note two files id_rsa and id_rsa.pub — corresponding to the private and public keys, respectively in ~/.ssh/ directory.

Private key id_rsa should never be shared.

View the contents of the public key

~/.ssh/id_rsa.pub

Copy public key to authorized_key

cat id_rsa.pub >> ~/.ssh/authorized_key

View the contents of the private key

~/.ssh/id_rsa

Copy content of private key.

Go to Jenkins dashboard and select Manage Jenkins.

Under this section, select Credentials

Click on (global) domain of jenkins System store

Next, click on blue button + Add Credentials at top right

Choose from the dropdown, SSH Username with private key as our kind of global credential.

Enter private-key as its Description

Scroll down to Username field and enter root

Select radio button Enter directly for Private Key

Click on Add button on the right and paste the copied private key as new secret into input field.

Select Create to save credential.

Add Jenkins Slave Node

Next, we add js to agent pool.

Go back to Manage Jenkins and select 'Manage Nodes and Clouds`

Click on blue button + New Node

Enter node name as Slave-Node and select Type, Permanent Agent

Select Create

In the next page shown, fill in the following:

Number of executors ----------> 3

Root root directory ----------> /home/root/jenkins-slave/

Labels -----------------------> slave

Launch method ----------------> Launch agent via SSH

Host -------------------------> 192.168.10.9

Credentials ------------------> root (private-key)

Host Key Verification Strategy --------> Non Verifying Verification Strategy

Availability --------------------------> Keep this agent online as much as possible

Select Save

Next Steps

Setup Icinga stack for monitoring and alerting.

Enjoy!