clair-cicd

Repo Status = this repo is a WIP but starting to show some promise!

Clair, released by CoreOS in Nov '16, is a very effective tool for statically analyzing docker images and assessing images against known vulnerabilities. Integrating Clair into a CI/CD pipeline:

is complex (believe this is mostly a documentation challenge)
creates performance problems (building the Postgres vulnerabilities database is slow)
in and of itself is insufficient from a risk assessment point of view because once vulnerabilities are identified there's a lack of prescriptive guidance on how to act on the identified vulnerabilities

This repo was created to address the above problems.

Background

The roots of this repo center around the belief that:

services should be run in Docker containers and thus a CI/CD pipeline should be focused on the automated generation, assessment and ultimately deployment of Docker images
understanding and assessing the risk profile of services is important ie. security is important
risk is assessed differently for docker images that could find their way to production vs docker images that will only ever be used in development
Docker images should not be pushed to a Docker registry until their risk profile is understood (this is an important one)
Clair can be a very effective foundation for the automated assessment of Docker image vulnerabilities when inserted into the CI/CD pipeline
the CI/CD pipeline has to be fast. how fast? ideally < 5 minutes between code commit and automated (CD) deployment begins rolling out a change
there should be a clear division of responsibilities between those who create a docker image and those that determine the risk of vulnerabilities in a docker image
the risk assessment process must generate evidence which can be used to understand the risk assessment decision

Key Participants

service engineer - responsible for implementing a service that packaged in a docker container
security analyst - responsible for defining the

Key Concepts

docker image
vulnerabilities
static vulnerability analysis
vulnerability whitelist
service profile

How to Use

Getting Started

To get started with using clair-cicd to assess risk, a service engineer inserts a single line of code into a service's CI script which runs the shell script assess-image-risk.sh. Part of the CI script's responsibility is to build the docker image username/repo:tag and then push username/repo:tag to a docker registry. The single line of clair-cicd code should appear after username/repo:tag is built but before username/repo:tag is pushed to a docker registry.

In this simple case, assess-image-risk.sh returns a zero exit status if username/repo:tag contains no known vulnerabilities above a medium severity. If username/repo:tag contains any known vulnerabilities with a severity higher than medium assess-image-risk.sh returns a non-zero exit status and the build fails immediately ie. the build should fail before username/repo:tag is pushed to a docker registry.

curl -s -L https://raw.githubusercontent.com/simonsdave/clair-cicd/master/bin/assess-image-risk.sh | bash -s -- "username/repo:tag"

Adding a Vulnerability Whitelist

Consider this scenario:

a high severity vulnerability exists in a command line utility that's part of username/repo:tag
you have a 100% confidence that the utility will never be used

Following the steps described in the previous section would result in the CI process failing because the high severity vulnerability would be detected and assess-image-risk.sh would return a non-zero exit status. But this failure seems inappropriate given the knowledge that the command line tool with the vulnerability would never be used. Enter clair-cicd's whitelists.

Whitelists are json documents which allow security analysts to influence assess-image-risk.sh risk assessment. Whitelist expectations:

maintained by security analyst not service engineer
checked into source code control and appropriate change management processes are used to make changes (code reviews, feature branches, etc)

Adding a Service Profile

service engineer defines a service profile

How it Works

Assumptions/requirements:

bash, curl & jq are installed and available
docker is installed and running
clair-cicd was originally developed on Ubuntu 14.04 although all development and testing is now done on Ubuntu 16.04 so while things should still work on Ubuntu 14.04 they will certainly work on Ubuntu 16.04 - if you try using clair-cicd on other platforms the feedback on how things go would be greatly appreciated

There are 4 moving pieces:

assess-image-risk.sh is bash script which does the heavy lifting to co-ordinate the interaction of the 3 other moving pieces
Clair which is packaged inside the docker image quay.io/coreos/clair
Clair's vulnerability database which is packaged inside the docker image simonsdave/clair-database - a Travis Cron Job is used to rebuild simonsdave/clair-database daily to ensure the vulnerability database is kept current
a set of Python and Bash risk analysis scripts packaged in the simonsdave/clair-cicd-tools docker image

cji/clair-cicd