ar51an/unbound-dashboard

Unbound Dashboard In Grafana With Prometheus & Loki

unbound dashboard

           

Summary

🔸 Unbound dashboard in Grafana
🔸 Prometheus time series database
🔸 Unbound metrics exporter in Go
🔸 Log aggregation with Loki
🔸 Unbound setup is available at unbound-redis
🔸 Refer to info.md for dashboard details and release.md for changes

Specs:

Grafana	Prometheus	Loki	Go	OS	HW
11.1.0	2.42.0	3.1.0	1.21.5	raspios-bookworm-arm64-lite	Raspberry Pi 4 Model B

Steps

  🔸 Grafana ➜ Prometheus ➜ Unbound Exporter ➜ Loki ➜ Import Dashboard

❯ Grafana

• Download:
  There are 2 versions OSS and Enterprise. OSS version is more than enough. Enterprise version installs too many extra packages (like unattended-upgrades and more). Below cmd downloads Grafana OSS for arm64.

  wget https://dl.grafana.com/oss/release/grafana_11.1.0_arm64.deb

• Install:

  sudo apt install musl

  ℹ️ Note:
  musl package is required since Grafana 10.2.3

  sudo dpkg -i grafana_11.1.0_arm64.deb

  ℹ️ Note:
  A tweaked grafana.ini is available in the release. It reduces memory footprint, removes usage collection, stops calls to grafana server/repo and has few more optimizations. You can use grafana.ini either from the release or the default one. Default config is located at /etc/grafana/grafana.ini

• Start:

  sudo systemctl daemon-reload
sudo systemctl enable grafana-server
sudo systemctl start grafana-server

• UI:

  http://<RP-IP>:3000/
  Default user/pass ➟ admin/admin

❯ Prometheus

• Install:

  sudo apt install prometheus

• Config:
  Enable unbound-exporter scraping in prometheus. A trimmed down prometheus config, prometheus.yml is available in the release. Take a backup of existing prometheus.yml, if you are interested in the default config. Copy prometheus.yml from the release to /etc/prometheus/ dir.

  ℹ️ Note:
  Provided prometheus.yml has only unbound-exporter metric collection enabled. Default metric collection for node and prometheus exporters are removed. Scraping interval is set to 5m.

• Remove Node Exporter:
  Node exporter exports machine metrics. It is installed as part of prometheus pkg and runs as systemd service. It is not needed for unbound-dashboard. Unless you are already using it, remove node exporter. Below cmd will remove 8 node-exporter related pkgs.

  Remove:
  sudo apt --purge autoremove prometheus-node-exporter
  Disable scrape config:
  Provided prometheus.yml has node exporter scrapping config removed.

• UI:

  http://<RP-IP>:9090/

❯ Unbound Exporter

• I wrote my own exporter in Go. It is more efficient and tailored for this dashboard. A prebuilt binary (for arm64) is available in the release. Source code is available at unbound-exporter.

• Config:
  Modify Unbound config. Edit /etc/unbound/unbound.conf

  • Enable extended stats. Add option under server: tag

    extended-statistics: yes

  • Enable Unix domain socket for collecting stats. It is faster than default TCP. Add below options under remote-control: tag

    control-interface: "/var/run/unbound.sock"
control-use-cert: no

• Install:
  Copy unbound-exporter binary from the release to /usr/local/bin/ dir. Make sure it is under ownership of root and executable.

  Change ownership [If needed]:
  sudo chown root:root /usr/local/bin/unbound-exporter
  Make it executable [If needed]:
  sudo chmod +x /usr/local/bin/unbound-exporter

• Service:
  Create service to automatically run unbound-exporter at startup. Copy prometheus-unbound-exporter.service from the release to /etc/systemd/system/ dir. Make sure it is under the ownership of root. Enable and start the service.

  Change ownership [If needed]:
  sudo chown root:root /etc/systemd/system/prometheus-unbound-exporter.service
  Enable:
  sudo systemctl enable prometheus-unbound-exporter
  Start:
  sudo systemctl start prometheus-unbound-exporter
  Status:
  sudo systemctl status prometheus-unbound-exporter

  ℹ️ Note:
  Provided prometheus-unbound-exporter.service passes 2 paramaters. Blocklist file path and Unbound unix domain socket URI. Change them accordingly if you are using different path and name.

• Usage:

  unbound-exporter -h

  

❯ Loki

• Download:
  Download Loki and Promtail

  curl -O -L "https://github.com/grafana/loki/releases/download/v3.1.0/loki_3.1.0_arm64.deb"
curl -O -L "https://github.com/grafana/loki/releases/download/v3.1.0/promtail_3.1.0_arm64.deb"

• Install:

  sudo dpkg -i loki_3.1.0_arm64.deb
sudo dpkg -i promtail_3.1.0_arm64.deb

• Logging:
  Enable Unbound logging.

  • Edit /etc/unbound/unbound.conf. Add/Modify below options under server: tag

    log-replies: yes
log-tag-queryreply: yes
log-local-actions: yes
logfile: /var/log/unbound/unbound.log

    ℹ️ Note:
    Make sure verbosity: is set to 0.

  • Create log dir

    sudo mkdir /var/log/unbound
sudo chown unbound:unbound /var/log/unbound

  • Enable log rotation

    Copy file unbound from the release under logrotate dir to /etc/logrotate.d/ dir. Make sure it is under the ownership of root.

• Config:
  Replace /etc/loki/config.yml and /etc/promtail/config.yml with the config.yml files in the release under loki and promtail dirs respectively. Both should be under the ownership of root.

  ℹ️ Note:
  Provided loki config.yml is optimized to process large data set and metrics calculation for Unbound logs. It is tweaked after some thorough testing in Unbound logs specific dashboard with 6 panels. Default loki config can hardly handle 2 panels with small data set, it throws errors and timeouts with large data set and multiple requests.
  Provided promtail config.yml enables Unbound logs scrapping.

• Restart:

  sudo systemctl restart loki
sudo systemctl restart promtail

❯ Import Dashboard

• Open Grafana UI ➟ http://<RP-IP>:3000/

  • Click Data Sources under Configuration. Click Add data source select Prometheus. Add below options:

    Name ➟ Prometheus
    Default ➟ On
    Add URL ➟ http://localhost:9090
    Add Scrape interval ➟ 5m
    Hit ➟ Save & test

  • Click Data Sources under Configuration. Click Add data source select Loki. Add below options:

    Name ➟ Loki
    Add URL ➟ http://localhost:3100
    Add Maximum lines ➟ 100000
    Hit ➟ Save & test

  • Dashboard, unbound-dashboard.json is available in the release. Click Import under Dashboards. Click Upload JSON file. Select unbound-dashboard.json. Add below options:

    Folder ➟ Dashboards
    Select Prometheus ➟ Data Source
    Select Loki ➟ Data Source
    Hit ➟ Import

❯ ℹ️ Tips & Notes

• Grafana:
  How to ➟ Change grafana landing page to unbound dashboard & Switch between Dark (default) and Light theme.

  Open Grafana UI ➟ http://<RP-IP>:3000/
  Click Profile under top right icon
  Under Preferences select General/Unbound in Home Dashboard drop down
  Change theme in Interface theme drop down
  Hit Save

  There is an additional panel in the dashboard at the top right, not visible in the preview. It shows unbound-exporter status and may be beneficial. If you are not interested in that simply remove it. Screenshot below:

  

• Prometheus:
  How to ➟ Remove time series (metrics) collected by prometheus instantly for fresh start & Reduce prometheus journal logging.

  Enable admin API:
  sudo nano /etc/default/prometheus
  Add at the top: ARGS="--web.enable-admin-api --log.level=warn"
  Save & Exit

  To delete all metrics of specific exporter add job_name as last argument in delete cmd:
  Delete:
  curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={job="node"}'
  Purge:
  curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/clean_tombstones'
  Restart:
  sudo systemctl restart prometheus