An example project demonstrating the deployment of two LibreNMS Stateful Sets via Kubernetes on Minikube (Kubernetes running locally on a workstation).
Contains example Kubernetes YAML resource files (in the 'distributed/standalone' folder) and associated Kubernetes based Shell scripts (in the 'scripts' folder) to configure the environment and deploy a LibreNMS Stateful Set, both a standalone and distributed setup.
Ensure the following dependencies are already fulfilled on your host Linux/Windows/Mac Workstation/Laptop:
- The VirtualBox hypervisor has been installed.
- The kubectl command-line tool for Kubernetes has been installed.
- The Minikube tool for running Kubernetes locally has been installed.
- The Minikube cluster has been started, inside a local Virtual Machine, using the following command (also includes commands to check that kubectl is configured correctly to see the running minikube pod):
$ minikube start
$ kubectl get nodes
$ kubectl describe nodes
$ kubectl get services
- LibreNMS depends on MySQL/MariaDB, and to bootstrap a single MySQL database, execute the following:
$ cd scripts
$ ./deploy_mysql.sh
- To deploy the LibreNMS Service, execute the following:
$ ./deploy_standalone_librenms.sh
- Re-run the following command, until the librenms-0 pod has successfully started ("Status=Running"; usually takes about a minute).
$ kubectl get po -l app=librenms
- Execute the following command, to validate the LibreNMS installation.
$ kubectl exec -ti librenms-0 -- su -p librenms -c "cd /opt/librenms && php validate.php"
You should now have a LibreNMS Stateful Set initialised, backed by a MySQL deployment.
You can also view the the state of the deployed environment, via the Kubernetes dashboard, which can be launched in a browser with the following command: $ minikube dashboard
Use this section to prove:
- The LibreNMS web interface is working as intended.
- Devices can be added and polled.
- Data is retained even when the LibreNMS StatefulSet is removed and then re-created (by virtue of re-using the same Persistent Volume Claims).
Use minikube to find the LibreNMS URL, so you can connect to the web interface using your browser:
$ minikube service librenms --url
> http://192.168.99.100:31971
Out of the box, a single user is created during deployment:
username: admin
password: admin
email: test@example.com
You should see an empty LibreNMS dashboard.
To test if LibreNMS is able to poll devices, add a device through the web interface or use $ kubectl exec:
$ kubectl exec -ti librenms-0 -- su -p librenms -c "cd /opt/librenms && php addhost.php example.com public v2c"
Now wait for the every-5-minute poller cronjob to complete, and use either the web interface or $ kubectl exec to validate if the device has been polled:
$ kubectl exec -ti librenms-0 -- su -p librenms -c "cat /opt/librenms/logs/librenms.log"
To see if Persistent Volume Claims really are working, run a script to drop the Service & StatefulSet (thus stopping the librenms pod) and then a script to re-create them again:
$ ./delete_standalone_service.sh
$ ./recreate_standalone_service.sh
$ kubectl get po -l app=librenms
As before, keep re-running the last command above, until you can see that "librenms" pod and their containers have been successfully started again. Then use $ kubectl exec to validate the LibreNMS installation, and use your browser to check the web interface:
$ kubectl exec -ti librenms-0 -- su -p librenms -c "cd /opt/librenms && php validate.php"
$ minikube service librenms --url
> http://192.168.99.100:31971
You should see that the graphs of the device you have added earlier, are still present..
Run the following script to undeploy the LibreNMS Service & StatefulSet.
$ ./teardown_standalone.sh
$ ./teardown_mysql.sh
If you want, you can shutdown the Minikube virtual machine with the following command.
$ minikube stop
- LibreNMS depends on MySQL/MariaDB, and to bootstrap a single MySQL database, execute the following:
$ cd scripts
$ ./deploy_mysql.sh
- To deploy the LibreNMS Service, execute the following:
$ ./deploy_distributed_librenms.sh
- Re-run the following command, until the librenms-0 pod has successfully started ("Status=Running"; usually takes about a minute).
$ kubectl get po -l app=librenms
- Execute the following command, to validate the LibreNMS installation.
$ kubectl exec -ti librenms-0 -- su -p librenms -c "cd /opt/librenms && php validate.php"
$ kubectl exec -ti librenms-pollers-0 -- su -p librenms -c "cd /opt/librenms && php validate.php"
You should now have a LibreNMS Stateful Set initialised, backed by a MySQL deployment.
You can also view the the state of the deployed environment, via the Kubernetes dashboard, which can be launched in a browser with the following command: $ minikube dashboard
Use this section to prove:
- The LibreNMS web interface is working as intended.
- Devices can be added and polled.
- Data is retained even when the LibreNMS StatefulSet is removed and then re-created (by virtue of re-using the same Persistent Volume Claims).
Use minikube to find the LibreNMS URL, so you can connect to the web interface using your browser:
$ minikube service librenms --url
> http://192.168.99.100:31971
Out of the box, a single user is created during deployment:
username: admin
password: admin
email: test@example.com
You should see an empty LibreNMS dashboard.
To test if LibreNMS is able to poll devices, add a device through the web interface or use $ kubectl exec:
$ kubectl exec -ti librenms-0 -- su -p librenms -c "cd /opt/librenms && php addhost.php example.com public v2c"
Now wait for the every-5-minute poller cronjob to complete, and use either the web interface or $ kubectl exec to validate if the device has been polled:
$ kubectl exec -ti librenms-0 -- su -p librenms -c "cat /opt/librenms/logs/librenms.log"
To see if Persistent Volume Claims really are working, run a script to drop the Service & StatefulSet (thus stopping the librenms pod) and then a script to re-create them again:
$ ./delete_distributed_service.sh
$ ./recreate_distributed_service.sh
$ kubectl get po -l app=librenms
$ kubectl get po -l app=rrdcached
As before, keep re-running the last command above, until you can see that "librenms" pod and their containers have been successfully started again. Then use $ kubectl exec to validate the LibreNMS installation, and use your browser to check the web interface:
$ kubectl exec -ti librenms-0 -- su -p librenms -c "cd /opt/librenms && php validate.php"
$ minikube service librenms --url
> http://192.168.99.100:31971
You should see that the graphs of the device you have added earlier, are still present..
Run the following script to undeploy the LibreNMS Service & StatefulSet.
$ ./teardown_distributed.sh
$ ./teardown_mysql.sh
If you want, you can shutdown the Minikube virtual machine with the following command.
$ minikube stop
The Secret Object contains base64 encoded strings for the admin username, password and email address.
$ echo "YWRtaW4=" | base64 --decode
> admin
$ echo "dGVzdEBleGFtcGxlLmNvbQ==" | base64 --decode
> test@example.com
Create your own secrets by base64 encoding the items:
$ echo -n "admin" | base64
YWRtaW4=
$ echo -n "test@example.com" | base64
dGVzdEBleGFtcGxlLmNvbQ==
The ConfigMap Object contains additional LibreNMS configuration, which will be mounted at /opt/librenms/conf.d.
If you add additional key-value pairs to the ConfigMap, you'd also need to add them to the items volume.
The ConfigMap keys basically contain PHP snippets, which allows for easy modification of the default LibreNMS configuration. The Distrubuted ConfigMap Object is an example of multiple snippets in one ConfigMap.
To access the LibreNMS web interface, a Service Object is defined. This service targets the LibreNMS Pods based on the app: librenms selector, and ties port 80 of the Pod to port 31971 on the Minikube VM. The service is accessible outside of the Kubernetes cluster because type is set to NodePort. The nodePort is also used in the LibreNMS BASE_URL variable.
Other services, for example memcached in the distributed setup, is only available inside of the Kubernetes cluster. Port 11211 of the Pod is exposed, and can be used by other Pods.
You can access services that are exposed to the Minikube VM by using $ minikube service <name> --url:
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 1h
librenms NodePort 10.109.170.147 <none> 80:31971/TCP,443:32459/TCP 1h
memcached ClusterIP None <none> 11211/TCP 1h
mysql ClusterIP None <none> 3306/TCP 1h
rrdcached ClusterIP None <none> 42217/TCP 1h
$ minikube service librenms --url
http://192.168.99.100:31971
$ kubectl run -it --rm --restart=Never nslookup --image=busybox nslookup librenms
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: librenms
Address 1: 10.109.170.147 librenms.default.svc.cluster.local
$ kubectl run -it --rm --restart=Never nslookup --image=busybox nslookup memcached
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: memcached
Address 1: 172.17.0.5 memcached-0.memcached.default.svc.cluster.local
The StatefulSet Controller manages the deployment of the different Pods for the standalone or distributed setup.
Initially, the setup of LibreNMS will prepare the database and create an admin user, which is managed by the initContainer. The initContainer should finish gracefully before the LibreNMS container will be started. If the initContainer fails (for example, when MySQL is unavailable), the Pod will be restarted until it succeeds.
In the distributed setup, other services are also declared as StatefulSets, for example rrdcached, since it requires persistent storage. In retrospect, I think I might refactor memcached and pollers as Deployment Controllers, since those services do not require storage or bootstrapping, and should be stateless.
- Deployment of a LibreNMS on a local Minikube Kubernetes platform.
- Use of Kubernetes StatefulSets and PersistentVolumeClaims to ensure data is not lost when containers are recycled.
- Proper configuration of a LibreNMS StatefulSet for fault tolerance.
- Securing the LibreNMS installtion with SSL certificates.
- Disabling Transparent Huge Pages to improve performance (this is disabled by default in the Minikube host nodes).
- Disabling NUMA to improve performance.
- Controlling CPU & RAM resource allocation.
- Adding a replicated MySQL setup.
- TBD
- Run a Single-Instance Stateful Application, re-used the MySQL deployment.
- pkdone/minikube-mongodb-demo by pkdone, re-used script and README.md layout.