An MVP for running Prometheus in GKE

Kubernetes in Google Kubernetes Engine, running Prometheus 2.1.0, scraping Kubernetes, itself and data from the nodes in the cluster.

Now with Persistent Volume on Prometheus! (so you can delete the Prometheus pod without losing data..)

Now with Grafana 4.6.3! (You still have to configure the Datasources manually because datasources-as-code isn't coming until 5.0.0 which is in review, and until then you can do it with a curl to the API) Issue (+ curl/python-script solutions): Adding datasource without using the web gui, PR: Datasources as configuration

Inspired heavily by https://coreos.com/blog/monitoring-kubernetes-with-prometheus.html from August 03, 2016.

Requirements

kubectl, gcloud and (of course..) rights to create the RBAC resources in the GKE cluster

Setup (minimal instructions)

Have access to a GKE cluster with kubectl
Run setup.sh
Wait about 10-15 minutes before the loadbalancers start serving traffic
Configure Prometheus data source in Grafana
???
Profit!

Cleanup

Run teardown.sh

Setup (detailed instructions)

1. Create a cluster on GKE

$ gcloud container clusters create my-prometheus-cluster

2. Create the `RBAC` resources from the `rbac/` folder

$ kubectl create -f rbac/service-account-prometheus.yaml
$ kubectl create -f rbac/clusterrole-prometheus.yaml
$ kubectl create -f rbac/clusterrolebinding-prometheus.yaml

Optional: Create the Node Exporter daemonset (if you want node-metrics like CPU-usage, etc.)

$ kubectl create -f node-exporter/daemonset-node-exporter.yaml

3. Create the `configmap` that we're going to mount into the Prometheus `deployment`

$ kubectl create -f prometheus/configmap-prometheus.yaml

4. Create the Prometheus `deployment` (with a `persistent volume`)

$ kubectl create -f prometheus/pv-deployment-prometheus.yaml

"old" deployment, w/o persistent volume: (or swap the commented lines in setup.sh, teardown.sh)

$ kubectl create -f prometheus/deployment-prometheus.yaml

5. Create the Prometheus Service, exposing the deployment as a `NodePort` since GKE ingress' require this

$ kubectl create -f networking/service-prometheus.yaml

6. Create the `ingress` loadbalancer

$ kubectl create -f networking/ingress-prometheus.yaml

7. Plenty of patience and Bob's your uncle

Use the following command to find the address Prometheus is being served on,

$ kubectl get ingress prometheus-ingress

Optional: Create the Grafana resources from the grafana/ folder, (you still have to add the datasource, for URL use the ip from the Prometheus service e.g. http://10.59.253.76:9090, and Access: Proxy)

$ kubectl create -f grafana/deployment-grafana.yaml
$ kubectl create -f grafana/service-grafana.yaml
$ kubectl create -f grafana/ingress-grafana.yaml

Add patience, and use the following command to find the address Grafana is being served on,

$ kubectl get ingress grafana-ingress

Infrequently Asked Questions:

Q: Why?!

A: Because I wanted to run something in kubernetes, and Prometheus seemed like a great idea, because it'd allow me to visualize stuff running in my cluster!

Q: Why are you exposing Prometheus to the world?!

A: .. Well it's not ideal. But I got to set up a loadbalancer for something running in Kubernetes, and expose it to the world, and now I can quickly see what's happening in my demo cluster. I mean, set up some firewall rules and you're (sort of) golden!

Q: Why is the `retainPolicy` of the Prometheus `PV`, `Delete`?

Because the PV is created automatically by a PVC. If you need Retain either edit the PV after it's been created, or create a PV-spec.

Roadmap

Upgrade Grafana to v5 for more Config as Code!

amorphid/gke-prometheus-mvp