tendermint-exporter

tendermint-exporter is a Prometheus scraper that scrapes some data to monitor your node, specifically you can set up alerting if:

your app version does not match the latest on Github (can be useful to be notified on new releases)
your voting power is 0 for a validator node
your node is catching up
your node is not in sync with the reference node (like the foundation one)

How can I set it up?

First of all, you need to download the latest release from the releases page. After that, you should unzip it and you are ready to go:

wget <the link from the releases page>
tar xvfz <the filename you've just downloaded>
./tendermint-exporter <params>

That's not really interesting, what you probably want to do is to have it running in the background. For that, first of all, we have to copy the file to the system apps folder:

sudo cp ./tendermint-exporter /usr/bin

Then we need to create a systemd service for our app:

sudo nano /etc/systemd/system/tendermint-exporter.service

You can use this template (change the user to whatever user you want this to be executed from. It's advised to create a separate user for that instead of running it from root):

[Unit]
Description=Cosmos Exporter
After=network-online.target

[Service]
User=<username>
TimeoutStartSec=0
CPUWeight=95
IOWeight=95
ExecStart=tendermint-exporter --config <path to config>
Restart=always
RestartSec=2
LimitNOFILE=800000
KillSignal=SIGTERM

[Install]
WantedBy=multi-user.target

If you're using cosmovisor, consider adding the same set of env variables as in your cosmovisor's systemd file, otherwise fetching app version would crash.

Then we'll add this service to the autostart and run it:

sudo systemctl enable tendermint-exporter
sudo systemctl start tendermint-exporter
sudo systemctl status tendermint-exporter # validate it's running

If you need to, you can also see the logs of the process:

sudo journalctl -u tendermint-exporter -f --output cat

How can I scrape data from it?

Here's the example of the Prometheus config you can use for scraping data:

scrape-configs:
  - job_name: 'tendermint-exporter'
    scrape_interval: 10s
    static_configs:
      - targets: ['<your IP>:9500']

Then restart Prometheus and you're good to go!

How does it work?

It fetches some data from the local node, another node as a reference node (like https://rpc.cosmos.network:443 for Cosmos) and GitHub.

How can I configure it?

You can pass the artuments to the executable file to configure it. Here is the parameters list:

--listen-address - the address with port the node would listen to. For example, you can use it to redefine port or to make the exporter accessible from the outside by listening on 127.0.0.1. Defaults to :9500 (so it's accessible from the outside on port 9500)
--local-tendermint-rpc - local Tendermint RPC URL to query node stats. Defaults to http://localhost:26657
--remote-tendermint-rpc - remote Tendermint RPC URL to query node stats. Optional, if not provided, the exporter won't scrape data from the remote node.
--binary-path - path to a fullnode binary to query version from. It may fail if it's located in $GOPATH and the path is relative, so better to explicitly provide the absolute path. Optional, if not provided the exporter won't collect data on binary version. This won't work on chains where <binary-path> version --long --output json doesn't work or returns data not in JSON.
--github-org - GitHub organization name.
--github-repo - GitHub repository. This param and --github-org are used to specify the repository hosting the full node binary sources. If one or both of these are not provided, the exporter won't fetch the remote GitHub version.
--github-token - GitHub personal access token. You can get one here. If this isn't provided, GitHub only allows you to do 60 requests per hour, if you do more requests than that it'll start throwing errors, causing all requests to this exporter to fail.
--log-devel - logger level. Defaults to info. You can set it to debug to make it more verbose.

Additionally, you can pass a --config flag with a path to your config file (I use .toml, but anything supported by viper should work). Here's the example of such config:

binary-path = "/home/validator/go/bin/gaiad"
github-org = "cosmos"
github-repo = "gaia"
remote-tendermint-rpc = "https://rpc.cosmos.network:443"

How can I contribute?