/monitoring

The monitoring/alerting stack for SCE resources

MIT LicenseMIT

Monitoring

The SCE Monitoring stack. A diagram of how we aggregate Prometheus metrics into grafana is below:

Monitoring Architecture

How does this work?

This use of Grafana/Prometheus/Discord bot alerting is covered in this YouTube series

Setting things up

  • Create an ssh key that lives at the location ~/.ssh/id_ed25519. This key should allow for automatic access to the clark machine. To allow for automatic access, run:
ssh-copy-id -i ~/.ssh/id_ed25519 sce@10.31.5.15
  • Get a webhook url for your Discord server and create a .env file in this project's directory like below:
DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/SERVER_ID/OTHER_STUFF
  • Run the project with
docker-compose up -d

Note: if you are not planning to run grafana behind nginx, expose port 3000 in the yml file and remove the GF_SERVER_ROOT_URL entry in the environment section for the Grafana container.