Dockerized Salt Master v3004 Silicon

Dockerfile to build a Salt Project Master image for the Docker opensource container platform.

salt-master is installed inside the Docker image using git source as documented in the the official bootstrap documentation.

For other methods to install salt-master please refer to the Official Salt Project Installation Guide.

Installation
- Changelog
Quick Start
Configuration
Usage
Shell Access
Restart Services
References

Installation

Automated builds of the image are available on Dockerhub and is the recommended method of installation.

docker pull cdalvaro/docker-salt-master:3004

You can also pull the latest tag which is built from the repository HEAD

docker pull cdalvaro/docker-salt-master:latest

These images are also available from GitHub Container Registry:

docker pull ghcr.io/cdalvaro/docker-salt-master:latest

and from Quay.io:

docker pull quay.io/cdalvaro/docker-salt-master:latest

Alternatively, you can build the image locally using make command:

make release

Quick Start

The quickest way to get started is using docker-compose.

wget https://raw.githubusercontent.com/cdalvaro/docker-salt-master/master/docker-compose.yml

Start the docker-salt-master container with the docker-compose.yml file by executing:

docker-compose up --detach

Alternatively, you can manually launch the docker-salt-master container:

docker run --name salt_master --detach \
    --publish 4505:4505 --publish 4506:4506 \
    --env 'SALT_LOG_LEVEL=info' \
    --volume $(pwd)/roots/:/home/salt/data/srv/ \
    --volume $(pwd)/keys/:/home/salt/data/keys/ \
    cdalvaro/docker-salt-master:latest

Configuration

Custom Configuration

This image uses its own master.yml file to configure salt-master to run properly inside the container. However, you can still tune other configuration parameters to fit your needs by adding your configuration files into a config/ directory and mounting it into /home/salt/data/config/.

For example, you can customize the Reactor System by adding a reactor.conf file to config/:

# config/reactor.conf
reactor:                        # Master config section "reactor"
  - 'salt/minion/*/start':      # Match tag "salt/minion/*/start"
    - /srv/reactor/start.sls    # Things to do when a minion starts

Then, you have to add the start.sls file into your roots/reactor/ directory:

# roots/reactor/start.sls
highstate_run:
  local.state.apply:
    - tgt: {{ data['id'] }}

Finally, run your docker-salt-master instance mounting the required directories:

docker run --name salt_master -d \
    --publish 4505:4505 --publish 4506:4506 \
    --volume $(pwd)/roots/:/home/salt/data/srv/ \
    --volume $(pwd)/keys/:/home/salt/data/keys/ \
    --volume $(pwd)/config/:/home/salt/data/config/ \
    cdalvaro/docker-salt-master:latest

This image provides support for automatically restart salt-master when configuration files change. This support is disabled by default, but it can be enabled by setting the SALT_RESTART_MASTER_ON_CONFIG_CHANGE environment variable to true.

Custom States

In order to provide salt with your custom states you must mount the volume /home/salt/data/srv/ with your roots directory inside it.

Minion Keys

Minion keys can be added automatically on startup to docker-salt-master by mounting the volume /home/salt/data/keys and copying the minion keys inside keys/minions/ directory.

It is also important to know that, in order to keep your keys after removing the container, the keys directory must be mounted.

mkdir -p keys/minions
rsync root@minion1:/etc/salt/pki/minion/minion.pub keys/minions/minion1

docker run --name salt_master -d \
    --publish 4505:4505 --publish 4506:4506 \
    --env 'SALT_LOG_LEVEL=info' \
    --volume $(pwd)/roots/:/home/salt/data/srv/ \
    --volume $(pwd)/keys/:/home/salt/data/keys/ \
    --volume $(pwd)/config/:/home/salt/data/config/ \
    cdalvaro/docker-salt-master:latest

Also, you can set your docker-salt-master instance to autoaccept minions that match certain grains. To do that, add the autosign_grains.conf to your config directory:

# config/autosign_grains.conf
autosign_grains_dir: /home/salt/data/srv/autosign_grains

Then, inside roots/autosign_grains you can place a file named like the grain you want to match and fill it with the content to match. For example, if you want to autoaccept minions that belong to specific domains, you have to add the domain file with the domains you want to allow:

# roots/autosign_grains/domain
cdalvaro.io
cdalvaro.com

It is possible that you have to configure the minion to send the specific grains to the master in the minion config file:

autosign_grains:
  - domain

More info at: Salt Project - Autoaccept Minions From Grains

Master Signed Keys

It is possible to use signed master keys by establishing the environment variable SALT_MASTER_SIGN_PUBKEY to True.

docker run --name salt_stack --detach \
    --publish 4505:4505 --publish 4506:4506 \
    --env 'SALT_LOG_LEVEL=info' \
    --env 'SALT_MASTER_SIGN_PUBKEY=True' \
    --volume $(pwd)/roots/:/home/salt/data/srv/ \
    --volume $(pwd)/keys/:/home/salt/data/keys/ \
    cdalvaro/docker-salt-master:latest

The container will create the master_sign key and its signature. More information about how to configure the minion service can be found here.

Additionally, you can generate new keys by executing the following command:

docker run --name salt_stack -it --rm \
    --volume $(pwd)/keys/:/home/salt/data/keys/ \
    cdalvaro/docker-salt-master:latest \
    app:gen-signed-keys new_master_sign

The newly created keys will appear inside keys/generated/new_master_sign directory.

Salt API

You can enable salt-api service by setting env variable SALT_API_SERVICE_ENABLED to true.

A self-signed SSL certificate will be automatically generated and the following configuration will be added to the master configuration file:

rest_cherrypy:
  port: 8000
  ssl_crt: /etc/pki/tls/certs/docker-salt-master.crt
  ssl_key: /etc/pki/tls/certs/docker-salt-master.key

The container exposes port 8000 by default, although you can map this port to whatever port you like in your docker run command:

docker run --name salt_stack --detach \
    --publish 4505:4505 --publish 4506:4506 --publish 8000:8000 \
    --env 'SALT_API_SERVICE_ENABLED=true' \
    --env 'SALT_API_USER_PASS=4wesome-Pass0rd' \
    --volume $(pwd)/roots/:/home/salt/data/srv/ \
    --volume $(pwd)/keys/:/home/salt/data/keys/ \
    --volume $(pwd)/config/:/home/salt/data/config/ \
    cdalvaro/docker-salt-master:latest

If you choose using the docker-compose.yml to manage your salt-master instance, uncomment salt-api settings to enable and configure the service.

By default, user salt_api is created and you can set its password by setting the environment variable SALT_API_USER_PASS.

You can also change the salt-api username by setting SALT_API_USER. It is possible to disable this user by explicitly setting this variable to an empty string: SALT_API_USER='' if you are going to use an LDAP server.

As a security measure, if SALT_API_SERVICE_ENABLED is set to true and you don't disable SALT_API_USER, you'll be required to set SALT_API_USER_PASS. Otherwise initialization will fail and your Docker image won't work.

With all that set, you'll be able to provide your salt-api custom configuration by creating the salt-api.conf file inside your conf directory:

external_auth:
  pam:
    salt_api:
      - .*
      - "@runner"
      - "@wheel"
      - "@jobs"

More information is available in the following link: External Authentication System (eAuth).

Now you have your docker-salt-master docker image ready to accept external authentications and to connect external tools such as saltstack/pepper.

Salt Pepper

The pepper CLI script allows users to execute Salt commands from computers that are external to computers running the salt-master or salt-minion daemons as though they were running Salt locally

Installation:

pip3 install salt-pepper

Configuration

Then configure pepper by filling your ~/.pepperrc file with your salt-api credentials:

[main]
SALTAPI_URL=https://your.salt-master.hostname:8000/
SALTAPI_USER=salt_api
SALTAPI_PASS=4wesome-Pass0rd
SALTAPI_EAUTH=pam

Usage

Beging executing salt states with pepper:

pepper '*' test.ping

Host Mapping

Per default the container is configured to run salt-master as user and group salt with uid and gid 1000. From the host it appears as if the mounted data volumes are owned by the host's user/group 1000 and maybe leading to unfavorable effects.

Also the container processes seem to be executed as the host's user/group 1000. The container can be configured to map the uid and gid of git to different ids on host by passing the environment variables PUID and PGID. The following command maps the ids to the current user and group on the host.

docker run --name salt_stack -it --rm \
    --publish 4505:4505 --publish 4506:4506 \
    --env "PUID=$(id -u)" --env "PGID=$(id -g)" \
    --volume $(pwd)/roots/:/home/salt/data/srv/ \
    --volume $(pwd)/keys/:/home/salt/data/keys/ \
    cdalvaro/docker-salt-master:latest

Git Fileserver

This image uses GitPython and PyGit2 as gitfs backends to allow Salt to serve files from git repositories.

It can be enabled by adding gitfs to the fileserver_backend list (see Available Configuration Parameters), and configuring one or more repositories in gitfs_remotes.

GitPython

The default name for the ssh key is gitfs_ssh but it can be changed with the env variables SALT_GITFS_SSH_PRIVATE_KEY and SALT_GITFS_SSH_PUBLIC_KEY.

This keys must be placed inside /home/salt/data/keys directory.

PyGit2

You can create an ssh key for pygit2 with the following command:

ssh-keygen -f gitfs_ssh -C 'gitfs@example.com'

Place it wherever you want inside the container and specify its path with the configuration parameters: gitfs_pubkey and gitfs_privkey in your .conf file.

For example:

gitfs_provider: pygit2
gitfs_privkey: /home/salt/data/keys/gitfs/gitfs_ssh
gitfs_pubkey: /home/salt/data/keys/gitfs/gitfs_ssh.pub

Important Note

By default, this image has been tested with RSA 4096 ssh keys generated with ssh-keygen.

If you get the following error while using gitfs with pygit2

_pygit2.GitError: Failed to authenticate SSH session: Unable to send userauth-publickey request

you may have to recreate your ssh key adding the parameter: -m PEM:

ssh-keygen -m PEM -f gitfs_ssh -C 'gitfs@example.com'

3rd Party Formulas

You can add third party formulas to your configuration simply by adding them to your gitfs_remotes:

# fileserver.conf
fileserver_backend:
  - roots
  - gitfs

# gitfs.conf
gitfs_provider: pygit2
gitfs_remotes:
  - https://github.com/saltstack-formulas/apache-formula
  - https://github.com/aokiji/salt-formula-helm.git

This is the Salt recommended way of doing it, and you can go to the Git Fileserver section on this document if you need help configuring this service.

You can find a great set of formulas on the following GitHub repositories:

Although, as mention in Salt Project documentation, you are encouraged to fork desired formulas to avoid unexpected changes to your infrastructure.

However, sometimes you may need to load some formulas that are not available on a git repository and you want to have them separated from your main srv directory.

For that case, you can mount a volume containing all your third party formulas separated in subdirectories into /home/salt/data/3pfs/, and they will be automatically added to the master configuration when your container starts.

# 3pfs directory content
3pfs
├── custom-formula
├── golang-formula
└── vim-formula

docker run --name salt_stack -it --rm \
    --publish 4505:4505 --publish 4506:4506 \
    --env "PUID=$(id -u)" --env "PGID=$(id -g)" \
    --volume $(pwd)/roots/:/home/salt/data/srv/ \
    --volume $(pwd)/3pfs/:/home/salt/data/3pfs/ \
    --volume $(pwd)/keys/:/home/salt/data/keys/ \
    cdalvaro/docker-salt-master:latest

If you need to add more third party formulas, you can restart the container, or you can type the following command:

docker exec -it salt_stack /sbin/entrypoint.sh app:reload-3rd-formulas

file_roots base configuration file will be updated with current existing formulas and salt-master service will be restarted to reload the new configuration.

Logs

Salt logs are accessible by mounting the volume /home/salt/data/logs/.

Inside that directory you could find supervisor/ logs and salt/ logs:

docker run --name salt_master --detach \
    --publish 4505:4505 --publish 4506:4506 \
    --env 'SALT_LOG_LEVEL=info' \
    --volume $(pwd)/roots/:/home/salt/data/srv/ \
    --volume $(pwd)/keys/:/home/salt/data/keys/ \
    --volume $(pwd)/logs/:/home/salt/data/logs/ \
    cdalvaro/docker-salt-master:latest

Check Available Configuration Parameters section for configuring logrotate.

Healthcheck

This image includes a health check script: /usr/local/sbin/healthcheck (although it is disable by default). It is useful to check if the salt-master service is alive and responding.

If you are running this image under k8s, you can define a liveness command as explained here.

If you use docker-compose as your container orchestrator, you can add the following entries to your compose file:

version: "3.4"

services:
  master:
    container_name: salt_master
    image: cdalvaro/docker-salt-master:latest
    healthcheck:
      test: ["CMD", "/usr/local/sbin/healthcheck"]
      start_period: 30s

(More info available at compose file official documentation)

Or, if you launch your container with docker:

docker run --name salt_master --detach \
    --publish 4505:4505 --publish 4506:4506 \
    --health-cmd='/usr/local/sbin/healthcheck' \
    --health-start-period=30s \
    --env 'SALT_LOG_LEVEL=info' \
    --volume $(pwd)/roots/:/home/salt/data/srv/ \
    --volume $(pwd)/keys/:/home/salt/data/keys/ \
    --volume $(pwd)/logs/:/home/salt/data/logs/ \
    cdalvaro/docker-salt-master:latest

Then you can manually check this info by running the following command:

docker inspect --format "{{json .State.Health }}" salt_master | jq

Then, the output will be something similar to this:

{
  "Status": "healthy",
  "FailingStreak": 0,
  "Log": [
    {
      "Start": "2020-05-23T16:47:55.1046568Z",
      "End": "2020-05-23T16:48:02.3381442Z",
      "ExitCode": 0,
      "Output": "local:\n    True\n"
    }
  ]
}

Autoheal

If you run your docker-salt-master instance with healthcheck enabled, you can use willfarrell/autoheal image to restart your service when healthcheck fails:

docker run -d \
  --name autoheal \
  --restart=always \
  -e AUTOHEAL_CONTAINER_LABEL=all \
  -v /var/run/docker.sock:/var/run/docker.sock \
  willfarrell/autoheal

This container will watch your containers and restart your failing instances.

Available Configuration Parameters

Please refer the docker run command options for the --env-file flag where you can specify all required environment variables in a single file. This will save you from writing a potentially long docker run command. Alternatively you can use docker-compose.

Below you can find a list with the available options that can be used to customize your docker-salt-master installation.

Parameter	Description
`DEBUG`	Set this to `true` to enable entrypoint debugging.
`TIMEZONE` / `TZ`	Set the container timezone. Defaults to `UTC`. Values are expected to be in Canonical format. Example: `Europe/Madrid`. See the list of acceptable values.
`SALT_RESTART_MASTER_ON_CONFIG_CHANGE`	Set this to `true` to restart `salt-master` service when configuration files change. Default: `false`
`SALT_LOG_LEVEL`	The level of messages to send to the console. One of 'garbage', 'trace', 'debug', info', 'warning', 'error', 'critical'. Default: `warning`
`SALT_LOG_ROTATE_FREQUENCY`	Logrotate frequency for salt logs. Available options are 'daily', 'weekly', 'monthly', and 'yearly'. Default: `weekly`
`SALT_LOG_ROTATE_RETENTION`	Keep x files before deleting old log files. Defaults: `52`
`SALT_LEVEL_LOGFILE`	The level of messages to send to the log file. One of 'garbage', 'trace', 'debug', info', 'warning', 'error', 'critical'. Default: `warning`
`SALT_API_SERVICE_ENABLED`	Enable `salt-api` service. Default: `false`
`SALT_API_USER`	Set username for `salt-api` service. Default: `salt_api`
`SALT_API_USER_PASS`	`SALT_API_USER` password. Required if `SALT_API_SERVICE_ENBALED` is `true` and `SALT_API_USER` is not empty. Unset by default
`SALT_API_CERT_CN`	Common name in the request. Default: `localhost`
`SALT_MASTER_SIGN_PUBKEY`	Sign the master auth-replies with a cryptographic signature of the master's public key. Possible values: 'True' or 'False'. Default: `False`
`SALT_MASTER_USE_PUBKEY_SIGNATURE`	Instead of computing the signature for each auth-reply, use a pre-calculated signature. This option requires `SALT_MASTER_SIGN_PUBKEY` set to 'True'. Possible values: 'True' or 'False'. Default: `True`
`SALT_MASTER_SIGN_KEY_NAME`	The customizable name of the signing-key-pair without suffix. Default: `master_sign`
`SALT_MASTER_PUBKEY_SIGNATURE`	The name of the file in the master's pki-directory that holds the pre-calculated signature of the master's public-key. Default: `master_pubkey_signature`
`SALT_MASTER_ROOT_USER`	Forces `salt-master` to be runned as `root` instead of `salt`. Default: `False`
`SALT_GITFS_SSH_PRIVATE_KEY`	The name of the ssh private key for gitfs. Default: `gitfs_ssh`
`SALT_GITFS_SSH_PUBLIC_KEY`	The name of the ssh public key for gitfs. Default: `gitfs_ssh.pub`
`PUID`	Sets the uid for user `salt` to the specified uid. Default: `1000`.
`PGID`	Sets the gid for user `salt` to the specified gid. Default: `1000`.
`USERMAP_UID` (deprecated)	Same as `PUID`. Support will be removed in Salt 3005 release in favor of `PUID`.
`USERMAP_GID` (deprecated)	Same as `PGID`. Support will be removed in Salt 3005 release in favor of `PGID`.

Any parameter not listed in the above table and available in the following link, can be set by creating the directory config and adding into it a .conf file with the desired parameters:

mkdir config
cat > config/ports.conf << EOF
# The tcp port used by the publisher:
publish_port: 3505
# The port used by the communication interface.
ret_port: 3506
EOF

docker run --name salt_master -d \
    --publish 3505:3505 --publish 3506:3506 \
    --env 'SALT_LOG_LEVEL=info' \
    --volume $(pwd)/roots/:/home/salt/data/srv/ \
    --volume $(pwd)/keys/:/home/salt/data/keys/ \
    --volume $(pwd)/config/:/home/salt/data/config/ \
    cdalvaro/docker-salt-master:latest

Usage

To test which salt minions are listening the following command can be executed directly from the host machine:

docker exec -it salt_master salt '*' test.ping

Then, you can apply salt states to your minions:

docker exec -it salt_master salt '*' state.apply [state]

Shell Access

For debugging and maintenance purposes you may want access the container shell. If you are using docker version 1.3.0 or higher you can access a running container shell using docker exec command.

docker exec -it salt_master bash

Restart Services

You can restart containers services by running the following command:

docker exec -it salt_master entrypoint.sh app:restart [salt-service]

Where salt-service is one of: salt-master os salt-api (if SALT_API_SERVICE_ENABLED is set to true)

adrian-tiberius/docker-salt-master

Dockerized Salt Master v3004 Silicon

Table of Contents

Installation

Quick Start

Configuration

Custom Configuration

Custom States

Minion Keys

Master Signed Keys

Salt API

Salt Pepper

Installation:

Configuration

Usage

Host Mapping

Git Fileserver

GitPython

PyGit2

3rd Party Formulas

Logs

Healthcheck

Autoheal

Available Configuration Parameters

Usage

Shell Access

Restart Services

References