SAURON - VERSION AND DEPLOYMENT TRACKER

DESCRIPTION

Sauron, the all seeing eye! It is a service to generate automated reports and track migrations, changes and dependency versions for backend services also report on known CVE and security issues. A detailed description can be found in the internal RFC document.

COMPONENTS

Sauron Service is segregated into a few components that are described below:

Sauron Core: Sauron common library to be used by all sauron plugins. More details here
Sauron Service: Sauron entrypoint controller described in more details in this README
Sauron Plugin System: Sauron has an embedded plugin support that allows anyone to introduce its own logic without the need to rebuild, regenerate and stop/restart Sauron Service. More details in further sections.
Elasticsearch Cluster: Sauron uses Elasticsearch as a data storage. It is currently deployed in AWS and can be accessed here.
Kibana: Elasticsearch data can be explored using the built-in Kibana instance that can be accessed here.
Dependencytrack: Sauron includes an instance of Dependencytrack, a platform that allows organizations to identify and reduce risk from the use of third-party and open source components.

ARCHITECTURE OVERVIEW

RUNNING

Sauron can be deployed using Docker and Docker-Compose. It provides a Dockerfile that can be built using the following commands:

Build Sauron project using maven and ship it to a docker image:

make

Start Sauron stack and load the local plugin repository

docker-compose -f docker-compose.yml --compatibility up

This command will start three components:

elasticsearch: http://localhost:9200
kibana: http://localhost:5601
sauron-service: http://localhost:8080

Note: Since Sauron needs your maven, gradle, pip and nodejs configuration, the docker-compose.yml creates a volume for each configuration folder/file. If there is no configuration already created, please create one before running the command above. For more details please refer to docker-compose.yaml > Volumes

Docker

In order to run Sauron with you predefined configuration using docker, use the command below:

docker run \
    -e SPRING_CONFIG_LOCATION="/sauron/config/sauron-service.yml" \
    -e M2_HOME="/usr/share/maven" \
    -e SPRING_PROFILES_INCLUDE="local" \
    -e SPRING_CLOUD_CONFIG_ENABLED="false" \
    --mount type=bind,source=${PWD}/sauron-service/docker/config/sauron-service.yml,destination=/sauron/config/sauron-service.yml,readonly \
    --mount type=bind,source=${PWD}/sauron-service/plugins,destination=/sauron/plugins \
    --mount type=bind,source=${HOME}/.m2,destination=/root/.m2 \
    --mount type=bind,source=${HOME}/.gradle,destination=/root/.gradle \
    --mount type=bind,source=${HOME}/.pip,destination=/root/.pip \
    --mount type=bind,source=${HOME}/.npmrc,destination=/root/.npmrc \
    --mount type=bind,source=${HOME}/.ssh,destination=/root/.ssh,readonly \
    --name=sauron \
    -p 8080:8080 \
    ghcr.io/freenowtech/sauron/sauron-service:latest

If you need to use a specific version, please refer to Sauron Packages

CONFIGURATION

Sauron configuration can be set via application.properties file. The file path can be provided using the environment variable:

SPRING_CONFIG_LOCATION: it must contain the path pointing to the local file e.g. /path/to/config/my-properties.properties or /path/to/config/my-properties.yaml

Sauron supports Spring Cloud Config Server as a configuration provider. In order to set Config Server url, please use the environment variable below:

SPRING_CLOUD_CONFIG_URI: it must contain the URL pointing to the remote repository e.g. https://my-repository.com/my-config

For a Sauron configuration file example, please refer to sauron-service.yml

USAGE

Sauron Elasticsearch Index Template

Before start using Sauron, it is import to define the index template that will be used by Elasticsearch to create Sauron's index. It can be done by running the command line below:

elasticsearch/sauron-template.sh
elasticsearch/dependencies-template.sh

This template increases the number of fields, since usually the amount of dependencies and thus the amount of fields is huge, and some other minor optimizations.

Triggering Sauron Service

Sauron Service provides a REST api that allows one to trigger a new build. The detailed parameters can be found in its swagger documentation. Once a new build has been triggered, Sauron's pipeline will run applying all plugins pre-configured. The output will be stored in elasticsearch and can be queried afterwards using the Kibana Installation.

For more details, check in the next section how the plugin system works.

A build request example can be found below:

curl --verbose --location --request POST 'http://localhost:8080/api/v1/build' \
    --header 'Content-Type: application/json' \
    --data-raw '{
      "serviceName": "MyService",
      "repositoryUrl": "https://github.com/gazgeek/springboot-helloworld.git",
      "commitId": "41c7823dddbef43680a0726ccea0631519b9d3c1",
      "buildId": "2b8caa6c-8b55-4b57-b654-5a00d519f409",
      "owner": "Sauron",
      "eventTime": 1586962717770,
      "rollback": false,
      "returnCode": 0,
      "environment": "production",
      "release": "0.0.1",
      "user": "sauron-user",
      "dockerImage": "helloworld:0.0.1",
      "platform": "K8S"
    }'

Visualizing Sauron Data

As mentioned before, Sauron stack uses Elasticsearch + Kibana to respectively store and visualize data. Once the Sauron build is done, the information will be available and can be queried using Kibana.

In order to do that, a index pattern must be created in Kibana using this HOWTO.

Once it is done, you are able to use your data. Sauron provides a default Kibana Dashboard available here. Import it and voilà! Enjoy your data!

Kibana Dashboard Example

SAURON PLUGIN SYSTEM

Sauron has an embedded plugin system that allows anyone to insert its own business logic to extract information during the building/deploy process. It uses the PF4J which is a plugin framework written in Java, and provides a nice interface to implement an integration in your service.

During the startup Sauron loads all available plugins, and updates them every 5 minutes using the pre-defined plugin repository (Local or Artifactory). For more details please refer to sauron-service.yml.

Official Plugins

console Output

Prints the DataSet content to sysout.

data-sanitizer

Sanitizes the data before being processed by Sauron pipeline.

git-checkout

Checkout the project source code.

dependency-checker

Extracts dependencies information in CycloneDX and insert in DataSet.

dependencytrack-publisher

Publishes the dependencies to our internal Dependency Track instance.

maven-report

Retrieves information from pom.xml file.

elasticsearch-output

Stores the DataSet content into Elasticsearch.

protocw-checker

Checks whether a service is using protoc, and the protoc wrapper.

logs-report

Tells if your service has any logs on elasticsearch. You can configure indexes and time range to search for logs.

kubernetesapi-report

Allows Sauron to query Kubernetes API to retrieve the annotations and labels assigned to the resources, specified in the configuration, and stores the values into the DataSet.

sonarapi-report

Query Sonar API to retrieve your service related data, like Code Coverage, and stores into the DataSet.

thanosapi-report

Query Thanos API to retrieve your service related data(like RPM, Circuit Breaker), and stores into the DataSet.

readme-checker

Checks whether a service has or not a README.md file in its root folder.

bcrypt-passwordencoder-checker

Checks whether a service might be using Bcrypt to encode the passwords in the API

jaegerapi-report

Query Jaeger API to extract tracing information

cleanup

Cleanup Sauron workspace after each pipeline processing

Creating a New Sauron Plugin

Sauron provides a maven archetype in order to create a standard skeleton of a new plugin. To use that follow the steps below.

Install maven archetype

Checkout the sauron-plugin-archetype project:

$ git clone https://github.com/freenowtech/sauron.git

Install the archetype locally and add it to the local archetype catalog:

$ cd sauron-plugin-archetype
$ mvn clean install

Create new plugin skeleton

Generate the new plugin using the installed archetype:

$ mvn org.apache.maven.plugins:maven-archetype-plugin:3.1.2:generate
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------< org.apache.maven:standalone-pom >-------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] >>> maven-archetype-plugin:3.0.1:generate (default-cli) > generate-sources @ standalone-pom >>>
[INFO]
[INFO] <<< maven-archetype-plugin:3.0.1:generate (default-cli) < generate-sources @ standalone-pom <<<
[INFO]
[INFO]
[INFO] --- maven-archetype-plugin:3.0.1:generate (default-cli) @ standalone-pom ---
[INFO] Generating project in Interactive mode
[INFO] No archetype defined. Using maven-archetype-quickstart (org.apache.maven.archetypes:maven-archetype-quickstart:1.0)
Choose archetype:
...
Choose a number or apply filter (format: [groupId:]artifactId, case sensitive contains): : sauron-plugin-archetype
Choose archetype:
1:  remote -> com.freenow.sauron:sauron-plugin-archetype (Sauron Plugin Archetype)
Choose a number or apply filter (format: [groupId:]artifactId, case sensitive contains): : 1
[INFO] Using property: groupId = com.free-now.sauron.plugins
Define value for property 'artifactId': my-plugin
[INFO] Using property: version = 0.0.1-SNAPSHOT
[INFO] Using property: package = com.freenow.sauron.plugins
Define value for property 'className': MyPlugin
Confirm properties configuration:
groupId: com.free-now.sauron.plugins
artifactId: my-plugin
version: 0.0.1-SNAPSHOT
package: com.freenow.sauron.plugins
className: MyPlugin
 Y: :
[INFO] ----------------------------------------------------------------------------
[INFO] Using following parameters for creating project from Archetype: sauron-plugin-archetype:0.0.1-SNAPSHOT
[INFO] ----------------------------------------------------------------------------
[INFO] Parameter: groupId, Value: com.free-now.sauron.plugins
[INFO] Parameter: artifactId, Value: my-plugin
[INFO] Parameter: version, Value: 0.0.1-SNAPSHOT
[INFO] Parameter: package, Value: com.freenow.sauron.plugins
[INFO] Parameter: packageInPathFormat, Value: com/freenow/sauron/plugins
[INFO] Parameter: package, Value: com.freenow.sauron.plugins
[INFO] Parameter: version, Value: 0.0.1-SNAPSHOT
[INFO] Parameter: groupId, Value: com.free-now.sauron.plugins
[INFO] Parameter: className, Value: MyPlugin
[INFO] Parameter: artifactId, Value: my-plugin
[INFO] Project created from Archetype in dir: /Users/sergio/Documents/sauron/my-plugin
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  15.041 s
[INFO] Finished at: 2019-04-08T15:10:44+02:00
[INFO] ------------------------------------------------------------------------

It will generate a new maven project that follows the structure below:

Fill in the MyPlugin.java class with your desired logic:

package com.freenow.sauron.plugins;

import com.freenow.sauron.model.DataSet;
import com.freenow.sauron.properties.PluginsConfigurationProperties;
import org.pf4j.Extension;

@Extension
public class MyPlugin implements SauronExtension
{
    @Override
    public DataSet apply(PluginsConfigurationProperties properties, DataSet input)
    {
        // PLUGIN LOGIC

        return input;
    }
}

Configuring your new Plugin

In order to provide extra configuration to your plugin, refer to CONFIGURATION SECTION. After your configuration has been added, it will be available to be used by your plugin in PluginsConfigurationProperties object. See below an example of how you can use the configuration properties:

The following configuration provides an url to my-plugin plugin generated in step above:

sauron:
    plugins:
        my-plugin:
            url: https://my-plugin.com

So in order to retrieve this configuration uses the following java code:

@Extension
public class MyPlugin implements SauronExtension
{
    @Override
    public DataSet apply(PluginsConfigurationProperties properties, DataSet input)
    {
        properties.getPluginConfigurationProperty("my-plugin", "url").ifPresent(url -> System.out.println(url) );
        return input;
    }
}

Documenting your Plugin

Sauron Plugin Archetype provides a template of a README that must be filled to document what your plugin does, inputs, outputs and possible configuration.

Deploying your new Plugin

Once the developing and testing process has been done, you can deploy a new version of your plugin, using the pre-defined plugin repository (Local or Artifactory). For more details please refer to sauron-service.yml.

The plugin reloading process runs every 5 minutes. To force a reloading use the /api/v1/reload method.

Check Swagger Documentation for more information.

Using your new Plugin

Once your plugin has been deployed and loaded by Sauron Service you can use it in a pipeline which will be then applied to new deployments. Refer to Sauron Usage for detailed information.