Openwrt Collectd Graph Panel (Flux)

Grafana dashboard for Openwrt using Collectd and Influx (With Downsampling)

Introduction

In this instruction I will describe you how you can export your collectd statistics to InfluxDB and then display them in Grafana. This will allow you to get detailed statistics about your Openwrt (or any other system with collectd installed) performance. The graphs are almost realtime - the level of detail is 1 minute for last 24 hours. The statistics are downsampled what allows you to view information what was the system performance in the past (several month or years ago) while still keeping database size small. Of course level of details for statistics from one year ago is much lower (6 hours) that for the statistics for last 24 hours (1 minute). The solution that I describe has been created basing on Collectd to InfluxDB v2 with downsampling using Tasks.

Side note - if you want to have simple and working solution use: CGP. I am still using this and found it to be very versatile.

Prerequisities

Before you start you need to prepare and install following aplications on your server hardware (I am using Raspberrypi4 for this task):

- Influx 2.0 (probably it will work with version 1.8 also)
- Influx 2.0 CLI
- Telegraf
- Grafana
- Collectd

InfluxdB configuration

Create org my-org (if it already exist skip this step)
```
influx org create -n my-org
```

Download and Apply Influxdb template for Gociwd

wget https://raw.githubusercontent.com/mstojek/gociwd/main/gociwd-influx-template.yml
influx apply -o my-org -f gociwd-influx-template.yml

Create 'collectd' user with write access to bucket 'collectd/autogen' bucket
```
BUCKET=$(influx bucket list --hide-headers -n collectd/autogen | awk '{print $1}')
influx auth create --org my-org --description 'collectd' --read-bucket $BUCKET --write-bucket $BUCKET
```
Copy the created Token ID (called by me below InfluxDB-Token-For-Collectd-User), we will need it later for Telegraf config.

Create 'grafana' user with read access for the 'collectd/*' buckets

BUCKET_AUTOGEN=$(influx bucket list --hide-headers -n collectd/autogen | awk '{print $1}')
BUCKET_DAY=$(influx bucket list --hide-headers -n collectd/day | awk '{print $1}')
BUCKET_WEEK=$(influx bucket list --hide-headers -n collectd/week | awk '{print $1}')
BUCKET_MONTH=$(influx bucket list --hide-headers -n collectd/month | awk '{print $1}')
BUCKET_YEAR=$(influx bucket list --hide-headers -n collectd/year | awk '{print $1}')
BUCKET_TENYEAR=$(influx bucket list --hide-headers -n collectd/tenyear | awk '{print $1}')
BUCKET_HUNDYEAR=$(influx bucket list --hide-headers -n collectd/hundyear | awk '{print $1}')
influx auth create --org my-org --description 'grafana' --read-bucket $BUCKET_AUTOGEN --read-bucket $BUCKET_DAY --read-bucket $BUCKET_WEEK --read-bucket $BUCKET_MONTH --read-bucket $BUCKET_YEAR --read-bucket $BUCKET_TENYEAR  --read-bucket $BUCKET_HUNDYEAR

Copy the created Token ID (called by me Grafana-Token), we will need it later for Grafana config.

Setup Telegraf

Add stations counter to 'types.db' - Openwrt uses this type for counting WIFI connected stations
```
cat /usr/share/collectd/types.db
[...]
# OpenWRT
stations                value:GAUGE:0:256
```

Edit Telegraf configuration

cat /etc/telegraf/telegraf.d/collectd.conf
[[inputs.socket_listener]]
  service_address = "udp://:25827"
  data_format = "collectd"
  collectd_typesdb = ["/usr/share/collectd/types.db"]
  collectd_parse_multivalue = "split"

  [inputs.socket_listener.tags]
    bucket = "collectd"

[[outputs.influxdb_v2]]
  urls = ["http://<InfluxdDB-IP-address>:8086"]
  token = "<InfluxDB-Token-For-Collectd-User>"
  organization = "my-org"
  bucket = "collectd/autogen"

  [outputs.influxdb_v2.tagpass]
    bucket = ["collectd"]

Note that we use port 25827 as port 25826 is used by Collectd. You need also to fill proper 'InfluxdDB-IP-address' and 'InfluxDB-Token-For-Collectd-User'

Restart Telegraf service

sudo service telegraf restart

Set up Openwrt/Collectd to send statistics to Telegraf/InfluxDB

Login to Luci and go to Statistics->Setup->Output plugins->Network->Configure

In the Server Interface section provide Server Host as Telegraf-IP-address and Server Port as 25827

(In my example Telegraf-IP-address is 192.168.100.99)
If you prefer to edit config files instead of using Luci add following lines to your collectd conf file /etc/collectd/collectd.conf
```
LoadPlugin network
<Plugin network> 
Server "<Telegraf-IP-address>" "25827"
</Plugin>
```

Setup Grafana

Setup Flux Data Source

Open you Grafana address in browser (http://<Grafana-IP-address>:3000)

Login to Grafana

Navigate to Home->Connections->DataSources->AddDataSource
- Enter name for Data Source (e.g collectd)
- In section Query language Choose Flux
- In section HTTP Provide URL for your InfluxDB with proper port http://<InfluxdDB-IP-address>:8086
- In section Auth Uncheck Basic auth (nothing should be chosen)
- In section InfluxDB details As Organization provide my-org Provide Grafana-Token that we created above
- Click "Save&Test"
Import Grafana Dashboard
- Copy Dashboard Template file gociwd-grafana-template.json to your local hard disk
- Go to Grafana->Home->Dashboards->Import dashboard and Drag and Drop gociwd-grafana-template.json file to the Import Field.
- Go to Grafana->Home->Dashboards and open new Dashboard Openwrt Collectd Graph Panel (Flux)
Alternatively you can use different Grafana template file gociwd-grafana-template_AllwaysConnectNullValues.json see Connect Null Values for details

Example statistics

Load, Memory and CPU statistics
Interfaces Traffic data
Client Traffic data (Iptmon or Nlbwmon2Collectd required)
WIFI, Connections, DNS and DHCP leases statistics
More statistics can be easily added by creating new Charts in Grafana. By default all statistics collected by Collectd are send to InfluxDB.

Data aggregation

Data is aggeregated over time. Following aggregations are set:
- Daily - data resolution is 1 minute
- Weekly - data resolution is 5 minutes
- Monthly - data resolution is 30 minutes
- Yearly - data resolution is 6 hours
- 10 Year - data resolution is 1 day
- 100 Year - data resolution is 1 week Original data send from collectd is store in Collect/autogen bucket
Example charts:
- Hourly
- Daily
- Monthly
- 1 year ago in September

Chart parameters

We have following chart parameters that can be choosen form drop-down menu
- Hostname - we can collects statistics from several collectd instances on several hosts (e.g. if you have more than one OpenWRT router, or some Raspberrypi hosts etc...)
- Client - if you have Iptmon or Nlbwmon2Collectd installed you can filter Client Traffic statistics just for this client
- Interface - you can filter Interfaces Traffic data for choosen interface
- Wifi-Interface - you can filter Wifi data for choosen Wifi interface
- Aggregation Type - available options are: Min, Max, Mean
  - Explanation: in default configuration collectd sends statistics every 30 seconds (sometimes it could be 10 seconds or other). If you choose in my chart "monthly aggregation" (last month data) we can not have data resolution to be 30 seconds. Instead this resolution is changed to 5 minutes (to save disk space). In the 5 minutes period we have ten 30 seconds periods. Now we need to chose what the aggregate point should show - maximum value from those ten perdios, average value or maybe minimum? It depends on data type. For example for Traffic you might be interested in max value, while for Idle processor usage we might be wondering what was the minimum value. Here you can choose what is interesting for you
- Retention policy - this is just informative field you can not change this value, it shows what retention bucket has been chosed for displaying the data.

Connect Null Values

In the Grafana charts you have an option named Connect Null Values. I prepared two Grafana templates:

template where this value is set to Threshold <1h - this is file: gociwd-grafana-template.json

This seetting Properly displays short term charts

However the long term charts consist of dots that are not conected
template where this value is set to Always - this is file:gociwd-grafana-template_AllwaysConnectNullValues.json

This setting incorrectly displays short term charts

Note that between 9:10 and 13:30 nothing happened on the system, this chart should be "Null" or zero at this time. Instead Grafana connected last point from ~9:10 whit first point at ~13:30

However the long term data have pretty nice look
I do not have any solution for this issue.

References

Collectd Graph Panel - great tool to display collectd RRD files: CGP
My solution is an evolution of following system described at: Collectd to InfluxDB v2 with downsampling using Tasks

What I added is
- rewriting everything to Flux (not good decision as Flux is in maintenance mode now....)
- better handling of data downsampling
Small but important topic on how to properly calculate "total usage" counters: Add previousN and nextN to range
Very smart solution for dynamically choosing right bucket basing on timespan choosen in Grafana: http://wiki.webperfect.ch/index.php?title=Grafana:_Dynamic_Retentions_%28InfluxDB%29