This is still a work in progress and is my attempt at a unified build/deployment process to get various projects running in a Docker Swarm on my Raspberry Pi farm.
The projects include:
Each of the projects has had a docker build added to it with configurations for health checks and metrics added where possible. I
am using scripts to test how the multi-architecture images I need might be created before looking at the CI/CD process proper.
There are two varieties of scripts in the /bin directory which can be used:
- One set using the standard docker build and manifest commands to build multi-architecture images by doing builds on different architectures and then combining the results.
- Another set using the experimental docker buildx command to build multi-architecture images with a single command.
Scripts are also defined to build any golden images required, and an additional ingress proxy image to route requests to project's that serve content on request (websites basically). There are two versions of the proxy, a test image without any SSL security and without the upgrade-insecure-requests CSP setting to make testing the full environment easier, and a production environment version which creates the ingress proxy image with SSL security and also upgrades the CSP settings. The SSL certificates can then be injected via secrets or using a filesystem mount (see note on security).
After all the images are built, everything is then tied together in this project with a number of docker-compose files to orchestrate the applications using stacks. The external facing sites are configured with the test ingress proxy by default which can be overridden for production environments. You can run this test using:
docker-compose -f external-sites.yml up -d
The docker-compose files currently expose ports for monitoring so that I can test the various parts of the deployment using different docker hosts not running as a swarm. It should be possible to remove most of these at some point.
To deploy and run everything into the swarm you need:
docker stack deploy -c portainer-agent-stack.yml portainer
docker stack deploy -c monitoring.yml monitoring
docker stack deploy -c sync-gandi-dns.yml sync-gandi-dns
docker stack deploy -c dinosauria-bot.yml dinosauria-bot
docker stack deploy -c guinea-bot.yml guinea-bot
docker stack deploy -c external-sites.yml -c monitoring-nginx.yml -c external-sites-production.yml external-sites
The result should look something like this (ignoring any replicas):
A key part of making this all work on Raspberry Pi's is picking multi-architecture base images which have good 32-bit ARM (arm32v7, armv7, armhf) support, so that I can try to keep the docker files for each project as simple as possible.
I am using images based on alpine 3.11 for stability and consistency, and I was trying to avoid issues with an out-of-date libseccomp2 on the Raspberry Pi OS (32-bit/Raspbian/Buster) that I'm running:
For Java, I want to use version 11 and AdoptOpenJDK has better support for 32-bit ARM than
OpenJDK. However, the latest official Maven image for
AdoptOpenJDK is based on
adoptopenjdk:11-jdk-hotspot which in turn uses Ubuntu 20.04 (Focal Fossa) and has the same issue with libseccomp2 when run on
Raspberry Pi OS as mentioned earlier. Researching the issue further the simplest solution looks to be to just install an updated
backport of libseccomp2 using the following commands:
wget http://ftp.uk.debian.org/debian/pool/main/libs/libseccomp/libseccomp2_2.4.4-1~bpo10+1_armhf.deb
sudo dpkg -i libseccomp2_2.4.4-1~bpo10+1_armhf.deb
I would rather use these standard images than have to find or build a custom image, so I will be using this solution.
I have started to define golden images for re-use and as best practice.
I created this image to implement Certbot with an authenticator plugin for Gandi DNS services in an ARM/V7 image. This allows me to use Let's Encrypt for my certificate renewals.
I created this image for all my Nginx instances, it includes configuration files from my fork of Nginx Server Configs
and has a simple health check (using the convenient curl anti-pattern 😒), and the basic Nginx metrics exposed.
In no particular order:
- Build a CI/CD pipeline.
- Set proper replicas and resource limits.
- Push images to my own registry.
- Better image labelling & tagging.
- Implement Anchore analysis and scanning.
- Improve monitoring and health checks.
- Design some permanent Grafana dashboards.
- Build images via Docker Hub or GitHub Actions.
- Test reporting of CSP issues and other errors.
- Use Traefik instead of Nginx.
- Use Kubernetes.
- Configure using Ansible.
- Mirror on AWS/GCP/Azure.
API keys are injected into the required containers using Docker Swarm secrets. Site certificates and private keys are either injected into the ingress proxy container using secrets or via a filesystem mount as a shared volume, having been generated by the Certbot image.
Secrets can't be updated in Docker Swarm so updating the certificate's means creating new secrets and then spinning up a new ingress proxy container referencing the new secrets. Having the certificates in a shared volume is less secure but means the ingress proxy only needs to be restarted when they are updated.