Autoscaling Spring Boot with the Horizontal Pod Autoscaler and custom metrics on Kubernetes

Prerequisites

You should have minikube installed.

You should start minikube with at least 4GB of RAM:

minikube start \
  --memory 4096 \
  --extra-config=controller-manager.horizontal-pod-autoscaler-upscale-delay=1m \
  --extra-config=controller-manager.horizontal-pod-autoscaler-downscale-delay=2m \
  --extra-config=controller-manager.horizontal-pod-autoscaler-sync-period=10s

If you're using a pre-existing minikube instance, you can resize the VM by destroying it an recreating it. Just adding the --memory 4096 won't have any effect.

You should install jq — a lightweight and flexible command-line JSON processor.

You can find more info about jq on the official website.

Installing Custom Metrics Api

Make sure you are in the monitoring folder:

cd monitoring

Deploy the Metrics Server in the kube-system namespace:

kubectl create -f ./metrics-server

After one minute the metric-server starts reporting CPU and memory usage for nodes and pods.

View nodes metrics:

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" | jq .

View pods metrics:

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/pods" | jq .

Create the monitoring namespace:

kubectl create -f ./namespaces.yaml

Deploy Prometheus v2 in the monitoring namespace:

kubectl create -f ./prometheus

Deploy the Prometheus custom metrics API adapter:

kubectl create -f ./custom-metrics-api

List the custom metrics provided by Prometheus:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .

Get the FS usage for all the pods in the monitoring namespace:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/monitoring/pods/*/fs_usage_bytes" | jq .

Package the application

You package the application as a container with:

eval $(minikube docker-env)
docker build -t spring-boot-hpa .

Deploying the application

Deploy the application in Kubernetes with:

kubectl create -f kube/deployment.yaml

You can visit the application at http://minkube_ip:32000

You can send messages to the queue by visiting http://minkube_ip:32000/submit

You should be able to see the number of pending messages from http://minkube_ip:32000/metrics and from the custom metrics endpoint:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/messages" | jq .

Autoscaling workers

You can scale the application in proportion to the number of messages in the queue with the Horizontal Pod Autoscaler. You can deploy the HPA with:

kubectl create -f kube/hpa.yaml

You can send more traffic to the application with:

while true; do sleep 0.5; curl -s http://<minikube ip>:32000/submit; done

When the application can't cope with the number of icoming messages, the autoscaler increases the number of pods only every 3 minutes.

You may need to wait three minutes before you can see more pods joining the deployment with:

kubectl get pods

The autoscaler will remove pods from the deployment every 5 minutes.

You can inspect the event and triggers in the HPA with:

kubectl get hpa spring-boot-hpa

Appendix

Using the secrets checked in the repository to deploy the Prometheus adapter is not recommended.

You should generate your own secrets.

But before you do so, make sure you install cfssl - a command line tool and an HTTP API server for signing, verifying, and bundling TLS certificates

You can find more info about cfssl on the official website.

Once cfssl is installed you generate a new Kubernetes secret with:

make certs

You should redeploy the Prometheus adapter.