A simple build system

[namliz]

After a period of lots of little tweaks and modifications to nail down a bug the AMI count passed 100. That's getting a bit out of hand.

The workflow is to cd into the ansible directory of this repository, do packer build packer.json, wait 15 minutes for the process to run through, manually alter which AMI is referenced in the bootstrap scripts, deploy a new cluster, and manually poke around testing things out -- sometimes to only discover some minor thing is broken too late for comfort.

We're past the point of needing proper tests. And its beneficial at this point to automate some things to speed the process up.

So to get this going a simple build system is necessary. The general idea is to setup github hooks to kick off packer builds instead of doing it from a laptop. The 15 minutes will hopefully be cut down a bit, but even better is having a record of it happening.

An implementation detail

It seems wasteful to always keep a server up to listen for build hooks so the hook should instead trigger a lambda that will either forward it to the build server or notice its not up and turn it back on (via Autoscaling group of size 1 with scheduling to scale down to 0 when commits usually don't happen).

Getting the logs back out again

Remote Journal Logging - "Systemd journal can be configured to forward events to a remote server. Entries are forwarded including full metadata, and are stored in normal journal files, identically to locally generated logs. This can be used as an alternative or in addition to existing log forwarding solutions."

Where the logs are forwarded to and how they are persisted is being explored.

Benefits

• Speed up debugging
• Significantly reduce the need to SSH into individual members of a cluster
• Easy reference and sharing of logs
• Possibly write smoke tests against the logs