The basic idea of this prototype is:
- Instrument a Spring REST application to expose Prometheus metrics, useful to monitor HTTP calls
- Deploy one or more instances of such application in a Kubernetes cluster (currently a Microk8s cluster in a VM with Ubuntu)
- Set up a Prometheus server (Docker image seems the easiest way) configured to do service discovering through Kubernetes of the Spring applications and to scape metrics from them
- Deploy a stack based on Fluentbit and Loki to aggregate logs on Loki
- Configure Grafana dashboards and alerts for Prometheus and Loki data sources
-
follow steps in Microk8s docs https://microk8s.io/docs
sudo snap install microk8s --classic
sudo usermod -a -G microk8s $USER
sudo chown -f -R $USER ~/.kube
su - $USER
-
enable Microk8s add ons (should be optional; check if some add one is required for the prototype)
sudo microk8s enable dns metrics-server dashboard ingress
- run Gradle bootJar:
.\gradlew.bat clean bootJar
- build and tag the docker image:
docker build -t ntuveri/spring-monitoring .
where ntuveri is a valid account in Docker Hub - Docker Hub login with
docker login
with proper username and password - push the local docker image to the remote repository:
docker push ntuveri/spring-monitoring
docker run -d --name prometheus -p 9090:9090 -v $pwd/prometheus:/etc/prometheus prom/prometheus
docker run -d --name loki -p 3100:3100 -v $pwd/loki/loki-config.yaml:/etc/loki/local-config.yaml grafana/loki
docker run -d --name grafana -p 3000:3000 -v $pwd/grafana/conf/custom.ini:/etc/grafana/grafana.ini -v $pwd/grafana/provisioning:/etc/grafana/provisioning grafana/grafana
It should provision at startup data sources, dashboards, notifiers and SMTP configuration
Set up the Prometheus data source and install the following dashboards:
- add the Prometheus datasource (can be accessed trough host network with http://host.docker.internal:9090)
- add the JVM micrometer dashboard at https://grafana.com/grafana/dashboards/4701
Sample queries for Loki data source
{severity="ERROR"} | json | message_severity="ERROR" | line_format "{{.message_message}}" |~ "RuntimeException"
{severity="DEBUG"} | json pod_hash="kubernetes.labels[\"pod-template-hash\"]" | pod_hash="754b4596d8"
Sample query for alert based on Prometheus data source
sum by (app_kubernetes_io_name, instance) (rate({level="error"}[1h]))
Jaeger Ports / protocols (see https://www.jaegertracing.io/docs/1.26/getting-started/#all-in-one)
Port | Protocol | Component | Function |
---|---|---|---|
5775 | UDP | agent | accept zipkin.thrift over compact thrift protocol (deprecated, used by legacy clients only) |
6831 | UDP | agent | accept jaeger.thrift over compact thrift protocol |
6832 | UDP | agent | accept jaeger.thrift over binary thrift protocol |
5778 | HTTP | agent | serve configs |
16686 | HTTP | query | serve frontend |
14268 | HTTP | collector | accept jaeger.thrift directly from clients |
14250 | HTTP | collector | accept model.proto |
9411 | HTTP | collector | Zipkin compatible endpoint (optional) |
docker run -d --name jaeger -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 -p 5778:5778 -p 16686:16686 -p 9411:9411 jaegertracing/all-in-one
Url for scraping metrics of a pod exposed with a service through the Kubernetes API proxy
https://:16443/api/v1/namespaces/default/services/spring-prometheus-svc:8080/proxy/actuator
Url for scraping metrics of a pod through the Kubernetes API proxy
https://:16443/api/v1/namespaces/default/pods/spring-prometheus:8080/proxy/actuator
Default bearer token to access Kubernetes API server
token=$(microk8s kubectl -n kube-system get secret | grep default-token | cut -d " " -f1)
microk8s kubectl -n kube-system describe secret $token
- Set label application in metrics as expected by the JVM Micrometer Grafana Dashboard - Ok
- Test multiple replicas of the same app - Ok
- Test different app - Ok
- Test logback metrics
- Test memory kubernetes limits and JVM memory usage (try to replicate OutOfMemory exception) - Ok
- Add Kubernetes metrics and dashboard - Ok
- Alerts based on metrics - OK
- Alerts based on logs
- Set up an example of context, trace and span ids
- Set up Jaeger / Zipkin and explore traces
- Set up cross-links between logs and traces