A monitoring client for Promstack.
This project includes node-exporter, Cadvisor, nginx-exporter and Promtail.
First create a .env
file from .env.dist.
HOST_NAME
the name of the host where promstack-exporters is installed. This will be available as a label in scraped data.ENV
the env of the host where promstack-exporters is installed. This will be available as a label in promtail data.NODE_EXPORTER_PORT
node-exporter expose metrics for Prometheus at this port.CADVISOR_PORT
Cadvisor expose metrics for Prometheus at this port.NGINX_EXPORTER_PORT
nginx-exporter expose metrics for Prometheus at this port.NGINX_SCRAPE_URI
the nginx stub_status URL, see nginx-exporter sectionDOCKER_HOST_INTERNAL_IP
the internal IP of docker host, used by nginx-exporter to access host's nginx stub_page.LOKI_EXTERNAL_URL
Loki URL to be called by Promtail, see Promtail bellow.LOKI_USERNAME
Loki username for Promtail basic auth to Loki.LOKI_PASSWORD
Loki password for Promtail basic auth to Loki.
Then up the project with docker-compose:
git clone git@github.com:paaacman/promstack-exporters.git
cd promstack-exporters
docker-compose up -d
- node-exporter (host metrics collector, send data to Prometheus)
- cAdvisor (containers metrics collector, sent to Prometheus)
- nginx-exporter (collect nginx status and send it to Prometheus)
- promtail (get log files and send it to Loki)
Prometheus exporter for hardware and OS metrics exposed by *NIX kernels, written in Go with pluggable metric collectors.
Metrics are scraped by Prometheus, container needs to be accessible by Prometheus at port NODE_EXPORTER_PORT
.
See node-exporter documentation.
Cadvisor monitor containers, like docker containers.
Metrics are scraped by Prometheus, container needs to be accessible by Prometheus at port CADVISOR_PORT
.
nginx-exporter uses nginx status page to get data about nginx usage. Those data can then be scraped by Prometheus.
Metrics are scraped by Prometheus, container needs to be accessible by Prometheus at port NGINX_EXPORTER_PORT
.
Nginx stub_status page can be set with the NGINX_SCRAPE_URI
environment variable. To monitor host's nginx instance, nginx-exporter container have to access this URI on host network by adding this docker-compose configuration:
# docker-compose.yml
services:
nginxexporter:
...
extra_hosts:
- "${DOCKER_HOST_NAME:${DOCKER_HOST_INTERNAL_IP}"
...
See nginx-exporter for more information.
According to the doc:
Promtail is an agent which ships the contents of local logs to a private Loki instance or Grafana Cloud. It is usually deployed to every machine that has applications needed to be monitored.
It sends logs to your Loki instance (available in Promstack project) at the URL given in LOKI_EXTERNAL_URL
environment variable.
In this promstack-exporters project, Promtail is configured to log to Loki with basic auth, provided by a nginx reverse proxy. See nginx documentation for more information.
Adapt the scrape_configs
in promtail.yml to your needs.
See Promtail documentation for more information.
You can adapt or remove promtail the second scrape_config, depending on your needs.
# promtail/promtail.yml
...
- job_name: logs
static_configs:
...
To backup Promtail current state to /tmp/promtail_backup
, you can backup promtail_data
volume like this:
PROMTAIL_VOLUME_ID=$(docker volume ls -q | grep promtail_data)
docker run --rm --volume "$PROMTAIL_VOLUME_ID:/promtail" --volume "/tmp/promtail_backup:/backup" debian tar cpf "/backup/promtail-volume-backup.tar" -C "/" "promtail"
See this technic to restore the volume: https://github.com/fjh1997/docker_named_volume_backup.