HA Consul + Vault + Vault UI

This project is an example of using Consul, Vault, and Vault UI in a high availability (HA) configuration. Conveniently packaged as Docker services for provisioning via Docker Compose.

Features:

dnsmasq makes Consul DNS available to all containers. A secondary dnsmasq server is provided which grants HA to the DNS available to all containers. This allows consul-template to update DNS with zero DNS downtime. consul-template will create a lock to ensure it is not possible for both primary and secondary DNS servers to be down during DNS configuration updates as part of service discovery.
consul-template updates dnsmasq configuration and restarts dnsmasq when the configuration has changed (e.g. consul cluster size is increased on the fly). This makes consul DNS lookups HA.
Vault and Vault UI is registered via service discovery which is exposed via Consul DNS.
Vault UI makes use of Consul DNS to log into Vault. This means Vault UI does not necessarily need to know where Vault is because Consul service discovery takes care of that.

Prerequisites

Supplemental reading material:

Hitchhiker's guide to administering Vault

Getting started

Remove --scale vault=3 if you want to start one instance of Vault. docker-compose up -d would bring only Consul up in HA configuration.

docker-compose up --scale vault=3 -d

Initialize Vault.

docker-compose exec vault sh
vault operator init

Unseal Vault:

for key in <unseal_key1> <unseal_key2> <unseal_key3>; do vault operator unseal "${key}"; done

The unseal_keyX comes from the output of vault operator init. You'll need to repeat logging into (docker-compose exec) and unsealing the other two Vault instances.

docker-compose exec --index=2 vault
docker-compose exec --index=3 vault

Note: the Root Token will be used to log into the Vault UI.

Visit the web UI

In order to properly utilize consul DNS, your browser must be configured to use the SOCKS5 proxy listening on 127.0.0.1:1080.

Consul UI: http://consul.service.consul:8500/
Vault UI: http://vault-ui.service.consul:8000/

Experiment

With HA enabled, container instances of consul and vault can be terminated with minor disruptions.

Consul can be scaled up on the fly. consul-template will automatically update dnsmasq to include new services. dnsmasq will experience zero downtime.

docker-compose up --scale vault=3 --scale consul-worker=6 -d

To play with failover for killing consul instances, it is recommended to review fault tolerance for consul HA deployments.

Troubleshooting

DNS

Currently, output from the dnsmasq and dnsmasq-secondary servers are minimal. Verbosity of output can be increased for troubleshooting. Edit docker-compose.yml and add --log-queries to the dnsmasq command.

DNS client troubleshooting using Docker.

docker-compose run dns-troubleshoot

Using the dig command inside of the container.

# rely on the internal container DNS
dig consul.service.consul

# specify the dnsmasq hostname as the DNS server
dig @dnsmasq vault.service.consul

# reference vault DNS by tags
dig active.vault.service.consul
dig standby.vault.service.consul

Logs

View vault logs.

docker-compose logs vault

User docker exec to log into container names. It allows you to poke around the runtime of the container.

SOCKS5 proxy

Run a SOCKS5 proxy for use with your browser.

docker run --network docker-compose-ha-consul-vault-ui_internal --dns 172.16.238.2 --init -p 127.0.0.1:1080:1080 --rm serjs/go-socks5-proxy

Configure your browser to use SOCKS proxy at 127.0.0.1:1080.

License

MIT License

flaviod/docker-compose-ha-consul-vault-ui