/monitoring

Monitoring

This repo contains the monitoring setup for razor-go staker along with the host in which it is running.

We are using

  1. Victoria Metrics: To scrap prometheus metrics form exporters.
  2. Grafana: For dashboard on metrics
  3. loki: Aggregate logs and provide to grafana.
  4. promtail:An agent which ships the contents of local logs
  5. node-exporter: node-exporter provide host based metrics to Victoria scraper
  6. caAdvisor: caAdvisor provide containers based metrics to Victoria scraper
  7. Alertmanager: Via Alertmanager we can configure alert for our staker and host.

Prerequisites

  • You must have Docker and Docker Compose installed.
  • If you are running staker via docker and running on same host, then your staker should up and running with razor_network can refer here
  • You must expose razor-go metrics, can refer here

NOTE: razor_network is a docker based bridge network through which staker and monitoring stack communicate securely without exposing anything publicly.

Configuration

  • If your staker is running via binary, then

    1. In ./configs/prometheus.yml, replace "razor-go:2112" with "<private/public address of host>:2112"

  • For alerting you can add webhook in ./configs/alertmanager.yml, replace http://127.0.0.1:5001/ with your webhook URL. This will send you an alert in every 5min if metrics stops.

  • If you are running multiple stakers and want to monitor via single grafana dashboard

    1. You need to update ./config/prometheus.yml, add new target block where job_name: "razor-go" 
      - targets: ["<second-host-address>:2112"]
  labels:
    staker: "<staker-name>"
    2. Restart vmagent service docker-compose restart vmagent

Start monitoring stack

  • You can spin all agents at once via

    docker-compose up -d

    Can check the status of each service via

    docker-compose ps

  • You can open grafana at <private/public address of host>:3000, and get

    1. Can checkout Razor dashboard to monitor your staker.
    2. Insight of host metrics at Node Exporter Full dashboard.
    3. Containers Insight at Docker and OS metrics ( cadvisor, node_exporter ) dashboard.
    4. Can monitor alerts at Alertmanager dashboard.

NOTE: Configure firewall for port 3000 on your host to access grafana.

Troubleshoot Alerting

  1. In docker-compose.yml uncomment ports for alertmanager and vmalert.

  2. Configure firewall to allow access to ports 8880 and 9093.

  3. Check you get alerts on vmalert via http://<host_address>:8880/vmalert/alerts. vmalert is configured to scrap in every 2min.

  4. If you see alert in vmalert then look into alertmanager http://<host_address>:9093/#/alerts?, if you see alerts in there but you didn't get one then probably you need to check your weebhook.