IoT with Arduino and Grafana

This repository contains information on how to start with IoT using Arduino, Grafana, Prometheus and Grafana Loki. In this repository, we will use Grafana Cloud that includes hosted Grafana, Prometheus and Loki to remove the overhead of installing and maintaining these systems. All of these projects are open source and therefore if you want, you can run them by yourself.

Setting up and using Arduino IDE
Setting up and using Grafana Cloud
Sending Metrics
Sending Logs
Using Loki and Prometheus libraries
Examples of projects built with Arduino and Grafana Cloud

Setting up and using Arduino IDE

The open source Arduino IDE makes it easy to write code and upload it to the board. The Arduino software is easy to use for beginners, yet flexible enough for advanced users.

Installing

To start, we are going to install Arduino IDE, a platform that’s used to write and upload programs to Arduino compatible boards. Follow the installing instruction for your OS.

Set up Arduino IDE to support ESP32 board

This step is not necessary if Arduino hardware is used. If you decide to use ESP32 development boards (we are using ESP32 boards in most of the projects listed below), we will need to do couple additional steps:

If your OS won’t recognize the USB serial automatically, you’ll probably need to install CP210x USB to UART Bridge VCP Driver that lets your computer communicate with your development board.
Add ESP32 boards definition that adds support for the EP32 board. Go to Arduino > Preferences and add url https://dl.espressif.com/dl/package_esp32_index.json to the Additional Boards Manager URLs input. This open-source board definition adds support for programming ESP32 boards.

Then go to Tools > Boards > Boards Manager and add ESP32 board manager.

Now, in Tools > Boards, you are able to select your ESP32 board. This tells the Arduino IDE which profile and base libraries to use when compiling the firmware image, and how to flash it to the board.

Using Arduino IDE and uploading program

In the Arduino environment, the programs that we are writing are called sketches. Before uploading the sketches to your development board, make sure that:

Sketch can be compiled. Click on the Verify button on top of IDE. Verifying goes through your sketch, checks for errors and compiles it.
In the Tools > Board submenu, you’ve selected the correct type of development port
In the Tools > Port submenu, you’ve selected the new correct COM port
In the Tools > Board submenu, you’ve selected the correct type of development port

Sketches are then uploaded to development board using Upload button on top of IDE.

Setting up and using Grafana Cloud

As mentioned, we are going to be using Grafana Cloud — which comes with hosted Grafana, Grafana Loki, and Graphite — for our data storage and data visualization. The free tier comes with 10,000 series for Prometheus metrics and 50GB for logs in Loki, which is definitely more than enough for simple monitoring solutions.

Creating account

To start, we'll visit Grafana Cloud signup and create a new account. As soon as the account is all set up, we can see the portal with hosted Grafana, Loki, and Prometheus instances.

Sending Metrics

To setup your project for sennding metrics, click on the Send Metrics button

From the next page, first you will want to copy the Remote Write Endpoint address

And put the value in your config.h

#define GC_URL "prometheus-blocks-prod-us-central1.grafana.net"

Next copy the Username / Instance ID

#define GC_USER "137822"

Then create an API key by clicking Generate now

Give the key a meaingful name and choose MetricsPublisher

Then click Create API Key

Copy the value into the GC_PASS field

#define GC_PASS "eyJrIjoiMTkzNDFkMzM2YTNhZTRlNmE4ZDkyMjgzSTBhNGFiYTcwY2VjMzVjNiIsIm4iOiJlc3AzMi10ZXN0LTMiLCJpZCI6NDIwMDY1fQ=="

The final result should look something like:

#define WIFI_SSID     "your_ssid"
#define WIFI_PASSWORD "your_wifi_password"

#define GC_URL "prometheus-blocks-prod-us-central1.grafana.net"
#define GC_PATH "/api/prom/push"
#define GC_PORT 443
#define GC_USER "137822"
#define GC_PASS "eyJrIjoiMTkzNDFkMzM2YTNhZTRlNmE4ZDkyMjgzSTBhNGFiYTcwY2VjMzVjNiIsIm4iOiJlc3AzMi10ZXN0LTMiLCJpZCI6NDIwMDY1fQ=="

Sending Logs

Similar to metrics, click the Sending Logs button

We will take some values from the Data Source Settings section starting with the URL

#define GC_URL "logs-prod-us-central1.grafana.net"

Next grab the User

#define GC_USER "8435"

And now generate an API key by clicking Generate now

Create a meaninful name and choose MetricsPublisher

The final config.h should look similar to this

#define WIFI_SSID     "your_ssid"
#define WIFI_PASSWORD "your_wifi_pass"

#define GC_URL "logs-prod-us-central1.grafana.net"
#define GC_PORT 443
#define GC_PATH "/loki/api/v1/push"
#define GC_USER "8435"
#define GC_PASS "eyJrIjoiMTkzNDFkMzM2YTNhZTRlNmE4ZDkyMjgzSTBhNGFiYTcwY2VjMzVjNiIsIm4iOiJlc3AzMi10ZXN0LTMiLCJpZCI6NDIwMDY1fQ=="

Creating more API Keys

At this point, or anytime in the future, we can create the API keys for Loki and Prometheus, to publish metrics from the monitoring system to these databases. The API key can be created by clicking on API Keys in the navigation on the left side. Then we click on + Add API Key and create API keys.

Visualised data in Grafana

Hosted Loki & Prometheus instances are automatically added as data sources in your hosted Grafana. You can find them under the name grafanacloud-NAME-logs and grafanacloud-NAME-prom. You can use these data sources with data from monitoring solutions to create dahsboards or use them in Explore.

Using Loki and Prometheus libraries

We have created following Arduino libraries to make it easier for you to send data to Prometheus and Loki:

All libraries can be download and used directly trough your Arduino IDE Library manager.

Start with opening Arduino IDE. Then go to Tools > Manage Libraries...

Search for "Prometheus" and install PrometheusArduino, PromLokiTransport, SnappyProto (You may be prompted to install additional libraries, saying yes will be the easiest path but you can also install the necessary libraries manually, see below)

Search for "Loki" and install GrafanaLoki

For list of additional libraries that you might need to install, refer to PrometheusArduino dependencies and GrafanaLoki dependencies

For how to use PrometheusArduino and GrafanaLoki, refer to their READMEs and examples folder

Examples of projects built with Arduino and Grafana Cloud

We have created couple of DIY IoT projects that you can easily create by yourself. These project have README that contains all information needed for you to re-create them.

grafana/diy-iot