ATC doesn't scale

[ARMmaster17]

General Information

• This is a: feature request
• Captain version: main
• Proxmox version: v6.3-3

Issue Report

As of right now, ATC cannot scale to be more than one instance. To remedy this, ATC will need to be broken down into several microservices/components.

• Create a "cron"-style microservice that emits health check messages every X seconds.
• Cron service uses locking in an etcd-style DB so multiple services can check if health check cooldown has expired.
• Health checkers that read from cron message queue, perform health checks, and send messages to builders.
• Builders become their own microservice that reads from a queue from the health checkers.

May want to look into migrating to a microservice framework like kite or gizmo to better unify the implementation across all the apps.

Related issues

• If this is going to happen, this will block #4