/polkadot-prometheus-logger

PolkaDOT Prometheus logger with Promtail, Loki and Grafana

Substrate/PolkaDOT Prometheus scrapper with curl and custom logging to the journaled system to receive the metrics in Grafana with Promntail including new or changed fields.

A monitoring service on Grafana Cloud, Loki, Promtail, CURL

  • No need to install Grafana
  • No need to setup Loki
  • Serveless Setup and Installation
  • Full controll on what is send as logs
  • Substrate changes can be obtained easy
  • More....

Dashboard


installation

  • Create a bash script in "/usr/sbin/" and paste the code below, you can modify the scrapping time at sleep 15
  • Control what is logged to journald grep commnad at grep -v '#' | grep '_count' The current logged metric are all with _count
  • Change inside the script with your Substrate/Polkadio Prometheus PORT
  • http://localhost:/metrics
  • Or run polkadot daemon with a custom one by adding prometheus flags to your configuration file
  • Specify Prometheus exporter TCP Port Flag --prometheus-port
#!/bin/bash

set -euf -o pipefail
exec 1> >(logger -s -t $(basename $0)) 2>&1

while true;
do
    curl -sb -S http://localhost:<PORT>/metrics | grep -v '#' | grep '_count' 
    sleep 15
done
  • Create system unit to enable the scrip with "vi /etc/systemd/system/metrics.service"
[Unit]
Description=Prometheus Scrapper.
After=network.target 

[Service]
Type=simple 
ExecStart=/bin/bash /usr/sbin/metrics.sh

Restart=always
[Install]
WantedBy=multi-user.target
  • Reload the systemctl daemon
systemctl daemon-reload 
  • Enable the service
systemctl enable metrics.service
  • Start the service
systemctl start metrics.service
  • Check the service
systemctl status metrics.service

Promtail Installation

apt install sudo unzip -y \
&& sudo mkdir /opt/promtail \
&& cd /opt/promtail \
&& wget https://github.com/grafana/loki/releases/download/v2.2.1/promtail-linux-amd64.zip \
&& sudo unzip promtail-linux-amd64.zip

Promtail Configuration template, change the <DLOKI> URL

URL Can be taken from Grafana Cloud Pannel
server:
  http_listen_port: 0
  grpc_listen_port: 0
  log_level: error
clients:
  - url: <DLOKI>
positions:
  filename: /tmp/positions.yaml
  ignore_invalid_yaml: true
scrape_configs:
- job_name: journal
  journal:
    max_age: 60s
    labels:
      job: systemd-journal
  relabel_configs:
    - source_labels: ['__journal__systemd_unit']
      target_label: 'unit'
    - source_labels: ['__journal__hostname']
      target_label: 'hostname'
    - source_labels: ['__journal__host']
      target_label: 'host'
  pipeline_stages:
  # Get this logs
  - match:
      selector: '{job="systemd-journal", unit="metrics.service"}'
      stages:
      - regex:
          expression: '.*(<output>)'
      - labels:
          level:
          output: log
          stream: stream
          timestamp: time
      - timestamp:
          source: timestamp
          format: RFC3339Nano
      - output:
          source: output
  • Test the config with promtail
./promtail-linux-amd64 -config.file ./substrate-metrics.yaml --dry-run 

Grafana Cloud

  • Create Grafana Cloud Account and obtain your Loki URL with API key for Promtail
  • Choose the label "UNIT" in your log query and look for "metrics.service" that is runing on the machine
  • Query example:
{unit="metrics.service"} | regexp "(?P<metric>^[a-zA-Z0-9_]+)(?P<json>\\{([^}]+)\\})\\W(?P<counter>[0-9]+)"
  • Filtering on the queries with exact matching or regex
|= "polkadot_"
|= "substrate_"
|~ "substrate_.*kademlia"
  • Example
{unit="metrics.service"} | regexp "(?P<metric>^[a-zA-Z0-9_]+)(?P<json>\\{([^}]+)\\})\\W(?P<counter>[0-9]+)" |~ "substrate_.*kademlia"

Dashboard download link, Granaba.com###

Download link


For Support && Nominations

  • Display name for Validators. KSMNETWORK && KSMNETWORK-WEST

  • Riot @gtoocool:matrix.org

  • KUSAMA (KSM) Address H1bSKJxoxzxYRCdGQutVqFGeW7xU3AcN6vyEdZBU7Qb1rsZ

  • PolkaDOT (DOT) Address: 15FxvBFDd3X7H9qcMGqsiuvFYEg4D3mBoTA2LQufreysTHKA