Setup guide for Icinga2 with Grafana integration employing InfluxDB

This guide uses Debian 10 "Buster" as base

cat /etc/apt/sources.list

deb http://deb.debian.org/debian buster main
deb-src http://deb.debian.org/debian buster main

deb http://security.debian.org/ buster/updates main
deb-src http://security.debian.org/ buster/updates main

deb http://deb.debian.org/debian buster-updates main
deb-src http://deb.debian.org/debian buster-updates main

deb http://deb.debian.org/debian buster-backports main
deb-src http://deb.debian.org/debian buster-backports main

Setup Icinga2

Add source

wget -O - https://packages.icinga.com/icinga.key | apt-key add -

echo "deb http://packages.icinga.com/debian icinga-buster main" > /etc/apt/sources.list.d/icinga.list
echo "deb-src http://packages.icinga.com/debian icinga-buster main" >> /etc/apt/sources.list.d/icinga.list

Get packages

apt-get update
apt-get install icinga2 monitoring-plugins

Check service status

systemctl status icinga2.service

● icinga2.service - Icinga host/service/network monitoring system
   Loaded: loaded (/lib/systemd/system/icinga2.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2017-07-28 13:46:00 BST; 16s ago

Setup Icinga2 MySQL support

Get packages

apt-get install icinga2-ido-mysql mariadb-server mariadb-client

Questions during installation:

Enable Icinga 2's ido-mysql feature? No
Configure database for icinga2-ido-mysql with dbconfig-common? No

Secure MySQL installation

mysql_secure_installation

Create database and user

mysql -u root -p

CREATE DATABASE icinga2 CHARACTER SET utf8 COLLATE utf8_general_ci;
GRANT ALL ON icinga2.* TO 'icinga2'@'localhost' IDENTIFIED BY 'your-icinga2-pwd';

Import database schema

mysql -p icinga2 < /usr/share/icinga2-ido-mysql/schema/mysql.sql

Enable mysql module

icinga2 feature enable ido-mysql

Configure database access for icinga instance

vim /etc/icinga2/features-available/ido-mysql.conf

-  password = "",
-  host = "localhost",
+  password = "your-icinga2-pwd",
+  host = "127.0.0.1",

Setup Icinga2 web interface

Get packages

apt-get install apache2 icingacli icingaweb2 icingaweb2-module-monitoring \
                php php-intl php-imagick php-gd php-mysql php-curl php-mbstring

Set PHP default timezone

vim /etc/php/7.3/apache2/php.ini

-;date.timezone =
+date.timezone = Europe/Paris

systemctl restart apache2.service

Create database and user

mysql -u root -p

CREATE DATABASE icingaweb2 CHARACTER SET utf8 COLLATE utf8_general_ci;
GRANT ALL ON icingaweb2.* TO 'icingaweb2'@'localhost' IDENTIFIED BY 'your-icingaweb2-pwd';

Import database schema

mysql -p icingaweb2 < /usr/share/icingaweb2/etc/schema/mysql.schema.sql

Enable external command pipe

icinga2 feature enable command
systemctl restart icinga2.service

Create setup token

icingacli setup config directory --group icingaweb2
icingacli setup token create

Setup using web interface

http://your-public-host.name/icingaweb2/setup

Setup Token: Your Token
Next

Modules enabled: Monitoring
Next

Authentication Type: Database
Next

Resource Name: icingaweb_db
Database Type: MySQL
Host: 127.0.0.1
Port: 3306
Database Name: icingaweb2
Username: icingaweb2
Password: your-icingaweb2-pwd
Character Set: utf8
Use SSL: False
Next

Backend Name: icingaweb2
Next

Username: admin
Password: your-admin-pwd
Repeat password: your-admin-pwd
Next

Show Stacktraces: Yes
Show Application State Messages: Yes
User Preference Storage Type: Database
Logging Type: Syslog
Logging Level: Error
Application Prefix: icingaweb2
Facility: user
Next

Backend Name: icinga
Backend Type: IDO
Next

Resource Name: icinga_ido
Database Type: MySQL
Host: 127.0.0.1
Port: 3306
Database Name: icinga2
Username: icinga2
Password: your-icinga2-pwd
Character Set: utf8
Use SSL: False
Next

Transport Name: icinga2
Transport Type: Local Command File
Command File: /var/run/icinga2/cmd/icinga2.cmd
Next

Protected Custom Variables: *pw*,*pass*,community
Next

Setup graphing module Grafana

Add source

wget -O - https://packages.grafana.com/gpg.key | apt-key add -

echo "deb https://packages.grafana.com/oss/deb stable main" > /etc/apt/sources.list.d/grafana.list

Get packages

apt-get update
apt-get install grafana influxdb influxdb-client

Enable and start Grafana service

systemctl daemon-reload
systemctl enable grafana-server.service
systemctl start grafana-server.service

Create database and user

influx

CREATE DATABASE icinga2;
CREATE USER icinga2 WITH PASSWORD 'your-icinga2-pwd';

Send performance data to influx

icinga2 feature enable influxdb

vim /etc/icinga2/features-enabled/influxdb.conf

-  //host = "127.0.0.1"
-  //port = 8086
-  //database = "icinga2"
-  //flush_threshold = 1024
-  //flush_interval = 10s
-  //host_template = {
-  //  measurement = "$host.check_command$"
-  //  tags = {
-  //    hostname = "$host.name$"
-  //  }
-  //}
-  //service_template = {
-  //  measurement = "$service.check_command$"
-  //  tags = {
-  //    hostname = "$host.name$"
-  //    service = "$service.name$"
-  //  }
-  //}
+  host = "127.0.0.1"
+  port = 8086
+  database = "icinga2"
+  username = "icinga2"
+  password = "your-icinga2-pwd"
+  enable_send_thresholds = true
+  enable_send_metadata = true
+  flush_threshold = 1024
+  flush_interval = 10s
+  host_template = {
+    measurement = "$host.check_command$"
+    tags = {
+      hostname = "$host.name$"
+    }
+  }
+  service_template = {
+    measurement = "$service.check_command$"
+    tags = {
+      hostname = "$host.name$"
+      service = "$service.name$"
+    }
+  }

systemctl restart icinga2.service

Navigate to Grafana web interface

Url: http://your-public-host.name:3000
Username: admin
Password: admin

Create new Grafana datasource

Add Datasource: http://your-public-host.name:3000/datasources/new?gettingstarted

Name: InfluxDB
Type: InfluxDB
Default: Yes

Url: http://127.0.0.1:8086
Access: Server (Default)

Database: icinga2
User: icinga2
Password: your-icinga2-pwd

Import Grafana dashboard

Import Dashboard: http://your-public-host.name:3000/dashboard/import

Paste JSON from https://raw.githubusercontent.com/Mikesch-mp/icingaweb2-module-grafana/v1.1.8/dashboards/influxdb/base-metrics.json

Add Icinga Web Grafana module

cd /usr/share/icingaweb2/modules
wget -qO- https://github.com/Mikesch-mp/icingaweb2-module-grafana/archive/v1.3.5.tar.gz | tar xvz
mv icingaweb2-module-grafana-1.3.5 grafana
mkdir /etc/icingaweb2/modules/grafana

Configure Icinga Web Grafana module

Module configuriation (config.ini)

vim /etc/icingaweb2/modules/grafana/config.ini

Graph configuriation (graphs.ini)

vim /etc/icingaweb2/modules/grafana/graphs.ini

Enable anonymous access

vim /etc/grafana/grafana.ini

[auth.anonymous]
# enable anonymous access
-;enabled = false
+enabled = true

# set to true if you want to allow browsers to render Grafana in a <frame>, <iframe>, <embed> or <object>. default is false.
-;allow_embedding = false
+allow_embedding = true

systemctl restart grafana-server.service

Enable Icinga Web Grafana module

icingacli module enable grafana
chown -R www-data:icingaweb2 /etc/icingaweb2

Disable graphing for commands without graphs

Go to the service configuration and set the custom var grafana_graph_disable for all services, which have no Grafana graph: ssh, http, disk, and icinga.

vim /etc/icinga2/conf.d/services.conf

+  vars.grafana_graph_disable = true

systemctl restart icinga2.service

Setup SNMP daemon

Get packages

apt-get install snmp snmpd libsnmp-dev

Configure and setup user

systemctl stop snmpd.service
net-snmp-config --create-snmpv3-user -ro -A "your-secret-auth-pwd" -X "your-secret-priv-pwd" -a SHA -x AES snmp

vim /etc/snmp/snmpd.conf

- rocommunity public  default    -V systemonly
- rocommunity6 public  default   -V systemonly
+# rocommunity public  default    -V systemonly
+# rocommunity6 public  default   -V systemonly

- rouser   authOnlyUser
+# rouser   authOnlyUser

- trapsink     localhost public
+# trapsink     localhost public

-iquerySecName   internalUser
-rouser          internalUser
-defaultMonitors          yes
-linkUpDownNotifications  yes
+#iquerySecName   internalUser
+#rouser          internalUser
+#defaultMonitors          yes
+#linkUpDownNotifications  yes

systemctl start snmpd.service

Add custom check to SNMP

Script to monitor open file handles: check_open_files

apt-get install bc
mkdir -p /usr/local/lib/nagios/plugins

vim /usr/local/lib/nagios/plugins/check_open_files
chmod +x /usr/local/lib/nagios/plugins/check_open_files

vim /etc/snmp/snmpd.conf

- extend    test1   /bin/echo  Hello, world!
- extend-sh test2   echo Hello, world! ; echo Hi there ; exit 35
-#extend-sh test3   /bin/sh /tmp/shtest
+ extend open_files /usr/local/lib/nagios/plugins/check_open_files

systemctl restart snmpd.service

Test it

snmpwalk -v3 -l authPriv -u snmp -a SHA -A "your-secret-auth-pwd" -x AES -X "your-secret-priv-pwd" 127.0.0.1 .1.3.6.1.4.1.8072.1.3.2

Output:

...
iso.3.6.1.4.1.8072.1.3.2.4.1.2.10.111.112.101.110.95.102.105.108.101.115.1 = STRING: "0.27"
iso.3.6.1.4.1.8072.1.3.2.4.1.2.10.111.112.101.110.95.102.105.108.101.115.2 = STRING: "open=1088 max=401859"

Monitor SNMP hosts

Get packages

apt-get install nagios-snmp-plugins libdigest-hmac-perl

Configure Icinga and Icinga Web Grafana module

Hosts configuriation (hosts.conf)

vim /etc/icinga2/conf.d/hosts.conf

Services configuriation (services.conf)

vim /etc/icinga2/conf.d/services.conf

Groups configuriation (groups.conf)

vim /etc/icinga2/conf.d/groups.conf

Graph configuriation (graphs.ini)

vim /etc/icingaweb2/modules/grafana/graphs.ini

systemctl restart icinga2.service

Get a list of network interfaces

In case you experience issues with the network interface (Unknown interface eth0), use the following command to get a list of available network interfaces:

/usr/lib/nagios/plugins/check_snmp_int.pl -x your-secret-auth-pwd -p 161 -n eth0 -l snmp -X your-secret-priv-pwd -L sha,aes -H 127.0.0.1 -v

Import Grafana SNMP dashboard

Import Dashboard: http://your-public-host.name:3000/dashboard/import

Use the grafana-snmp-dashboard.

chrisss404/icinga2-influxdb-grafana

Setup guide for Icinga2 with Grafana integration employing InfluxDB

This guide uses Debian 10 "Buster" as base

Setup Icinga2

Add source

Get packages

Check service status

Setup Icinga2 MySQL support

Get packages

Secure MySQL installation

Create database and user

Import database schema

Enable mysql module

Configure database access for icinga instance

Setup Icinga2 web interface

Get packages

Set PHP default timezone

Create database and user

Import database schema

Enable external command pipe

Create setup token

Setup using web interface

Setup graphing module Grafana

Add source

Get packages

Enable and start Grafana service

Create database and user

Send performance data to influx

Navigate to Grafana web interface

Create new Grafana datasource

Import Grafana dashboard

Add Icinga Web Grafana module

Configure Icinga Web Grafana module

Enable anonymous access

Enable Icinga Web Grafana module

Disable graphing for commands without graphs

Setup SNMP daemon

Get packages

Configure and setup user

Add custom check to SNMP

Test it

Monitor SNMP hosts

Get packages

Configure Icinga and Icinga Web Grafana module

Get a list of network interfaces

Import Grafana SNMP dashboard

Congratulations you successfully completed this guide