Motivation | Metrics transformation | License
A Docker image on which Prometheus can scrape Kubernetes metrics provided by metrics-server. The image can run directly in a k8s cluster where Prometheus can use it as a target.
metrics-server seems to be the successor of heapster for Kubernetes monitoring. However, metrics-server currently only provides its metrics in JSON format via the Kubernetes API server.
Prometheus on the other hand expects a special text-based format. So in order for Prometheus to scrape those metrics, they must be transparently transformed from JSON to its own format on every request.
Other than metrics-server itself, this Docker container provides additional metrics metadata that are
retrieved via python kubernetes client
API calls and included in the Prometheus output.
- Prometheus scrapes the Docker container on
:9100/metrics
- Inside the Docker container uwsgi is proxying the request to kube proxy
- The Python kubernetes client will be created and requests the k8s API
- The API replies with JSON formatted metrics provided by metrics-server
- uwsgi calls transform.py
- transform.py rewrites the JSON into Prometheus readable output and hands the result back to uwsgi
- uwsgi sends the final response back
Simply run the Docker image in a cluster with a mounted token for a ServiceAccount in /var/run/secrets/kubernetes.io/serviceaccount/token
(needs access to call the API)
prometheus.yml
:
scrape_configs:
- job_name: 'kubernetes'
scrape_interval: '15s'
metrics_path: '/metrics'
static_configs:
- targets:
- <DOCKER_IP_ADDRESS>:9100
metrics-server provices metrics in the following format:
{
"kind": "PodMetricsList",
"apiVersion": "metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/metrics.k8s.io/v1beta1/pods"
},
"items": [
{
"metadata": {
"name": "etcd-server-events-abc",
"namespace": "kube-system",
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/etcd-server-events-ip-10-30-78-99.eu-central-1.compute.internal",
"creationTimestamp": "2018-08-20T03:19:05Z"
},
"timestamp": "2018-08-20T03:19:00Z",
"window": "1m0s",
"containers": [
{
"name": "etcd-container",
"usage": {
"cpu": "7m",
"memory": "125448Ki"
}
}
]
},
]
}
metrics-server-prom transforms it to the following format:
Note: Additional metadata (node
and ip
) have been added.
# HELP kube_metrics_server_pod_cpu The CPU time of a pod in seconds.
# TYPE kube_metrics_server_pod_cpu gauge
kube_metrics_server_pod_cpu{node="ip-10-30-78-99.eu-central-1.compute.internal",pod="etcd-server-events-abc",ip="10.30.62.138",container="etcd-container",namespace="kube-system"} 420
# HELP kube_metrics_server_pod_mem The memory of a pod in KiloBytes.
# TYPE kube_metrics_server_pod_mem gauge
kube_metrics_server_pod_mem{node="ip-10-30-78-99.eu-central-1.compute.internal",pod="etcd-server-events-abc",ip="10.30.62.138",container="etcd-container",namespace="kube-system"} 128475136
Copyright (c) 2018 cytopia