pyrra: A Go repository from Rohlik

Making SLOs with Prometheus manageable, accessible, and easy to use for everyone!

Dashboards to visualize SLOs in Grafana:

Watch the 5min lightning talk at Prometheus Day 2022:

Features

Support for Kubernetes, Docker, and reading from the filesystem
Alerting: Generates 4 Multi Burn Rate Alerts with different severity
Page listing all Service Level Objectives
- Search through names and labels
- Sorted by remaining error budget to see the worst ones quickly
- All columns sortable
- View and hide individual columns
- Clicking on labels to filter SLOs that contain the label
- Tool-tips when hovering for extra context
Page with details for a Service Level Objective
- Objective, Availability, Error Budget highlighted as 3 most important numbers
- Graph to see how the error budget develops over time
- Time range picker to change graphs
- Switch between absolute and relative chart scales
- Request, Errors, Duration (RED) graphs for the underlying service
- Multi Burn Rate Alerts overview table
Caching of Prometheus query results
Thanos: Disabling of partial responses and downsampling to 5m and 1h
connect-go and connect-web generate protobuf APIs
Grafana dashboard via --generic-rules generation

Feedback & Support

If you have any feedback, please open a discussion in the GitHub Discussions of this project.
We would love to learn what you think!

Demo

Check out our live demo on demo.pyrra.dev!
Grafana dashboards are available as demo on demo.pyrra.dev/grafana!

Feel free to give it a try there!

How It Works

There are three components of Pyrra, all of which work through a single binary:

The UI displays SLOs, error budgets, burn rates, etc.
The API delivers information about SLOs from a backend (like Kubernetes) to the UI.
A backend watches for new SLO objects and then creates Prometheus recording rules for each.
- For Kubernetes, there is a Kubernetes Operator available
- For everything else, there is a filesystem-based Operator available

For the backend/operator to do its work, an SLO object has to be provided in YAML-format:

apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelObjective
metadata:
  name: pyrra-api-errors
  namespace: monitoring
  labels:
    prometheus: k8s
    role: alert-rules
    pyrra.dev/team: operations # Any labels prefixed with 'pyrra.dev/' will be propagated as Prometheus labels, while stripping the prefix.
spec:
  target: "99"
  window: 2w
  description: Pyrra's API requests and response errors over time grouped by route.
  indicator:
    ratio:
      errors:
        metric: http_requests_total{job="pyrra",code=~"5.."}
      total:
        metric: http_requests_total{job="pyrra"}
      grouping:
        - route

Depending on your mode of operation, this information is provided through an object in Kubernetes, or read from a static file.

In order to calculate error budget burn rates, Pyrra will then proceed to create Prometheus recording rules for each SLO.

The following rules would be created for the above example:

http_requests:increase2w

http_requests:burnrate3m
http_requests:burnrate15m
http_requests:burnrate30m
http_requests:burnrate1h
http_requests:burnrate3h
http_requests:burnrate12h
http_requests:burnrate2d

The recording rules names are based on the originally provided metric. The recording rules contain the necessary labels to uniquely identify the recording rules in case there are multiple ones available.

Running inside a Kubernetes cluster

An example for this mode of operation can be found in examples/kubernetes.

Here two deployments are needed: one for the API / UI and one for the operator. For the first deployment, start the binary with the api argument.

When starting the binary with the kubernetes argument, the service will watch the apiserver for ServiceLevelObjectives. Once a new SLO is picked up, Pyrra will create PrometheusRule objects that are automatically picked up by the Prometheus Operator.

If you're unable to run the Prometheus Operator inside your cluster, you can add the --config-map-mode=true flag after the kubernetes argument. This will save each recording rule in a separate ConfigMap.

Applying YAML

This repository contains generated YAML files in the examples/kubernetes/manifests folder. You can use the following commands to deploy them to a cluster right away.

kubectl apply --server-side -f ./example/kubernetes/manifests/setup
kubectl apply --server-side -f ./example/kubernetes/manifests
kubectl apply --server-side -f ./example/kubernetes/manifests/slos

Applying YAML and validating webhooks via cert-manager

This repository contains more generated YAML files in the examples/kubernetes/manifests-webhook folder.

This example deployment additionally applies and self-sign Issuer and requests a certificate via cert-manager, so that the Kubernetes APIServer can connect to Pyrra to validate any configuration object before applying it to the cluster.

kubectl apply --server-side -f ./example/kubernetes/manifests-webhook/setup
kubectl apply --server-side -f ./example/kubernetes/manifests-webhook
kubectl apply --server-side -f ./example/kubernetes/manifests-webhook/slos

kube-prometheus

The underlying jsonnet code is imported by the kube-prometheus project. If you want to install an entire monitoring stack including Pyrra we highly recommend using kube-prometheus.

Install with Helm

Thanks to @rlex there is a Helm chart for deploying Pyrra too.

Running inside Docker / Filesystem

An example for this mode of operation can be found in examples/docker-compose.

You can easily start Pyrra on its own via the provided Docker image:

docker pull ghcr.io/pyrra-dev/pyrra:v0.7.0

When running Pyrra outside of Kubernetes, the SLO object can be provided through a YAML file read from the file system. For this, one container or binary needs to be started with the api argument and the reconciler with the filesystem argument.

Here, Pyrra will save the generated recording rules to disk where they can be picked up by a Prometheus instance. While running Pyrra on its own works, there won't be any SLO configured, nor will there be any data from a Prometheus to work with. It's designed to work alongside a Prometheus.

Tech Stack

Client: TypeScript with React, Bootstrap, and uPlot.

Server: Go with libraries such as: chi, ristretto, xxhash, client-go.

Generated protobuf APIs with connect-go for Go and connect-web for TypeScript.

Roadmap

Best to check the Projects board and if you cannot find what you're looking for feel free to open an issue!

Contributing

Contributions are always welcome!

See CONTRIBUTING.md for ways to get started.

Please adhere to this project's code of conduct.

Maintainers

Name	Area	GitHub	Twitter	Company
Nadine Vehling	UX/UI	@nadinevehling	@nadinevehling	Grafana Labs
Matthias Loibl	Engineering	@metalmatze	@metalmatze	Polar Signals

We are mostly maintaining Pyrra in our free time.

Acknowledgements

@aditya-konarde, @brancz, @cbrgm, @codesome, @ekeih, @guusvw, @jzelinskie, @kakkoyun, @lilic, @markusressel, @morremeyer, @mxinden, @numbleroot, @paulfantom, @RiRa12621, @tboerger, and Maria Franke.

While we were working on Pyrra in private these amazing people helped us with a look of feedback and some even took an extra hour for a in-depth testing! Thank you all so much!

Additionally, @metalmatze would like to thank Polar Signals for allowing us to work on this project in his 20% time.

FAQ

Why not use Grafana in this particular use case?

Right now we could have used Grafana indeed. In upcoming releases, we plan to add more interactive features to give you better context when coming up with new SLOs. This is something we couldn't do with Grafana.

Do I still need Grafana?

Yes, Grafana is an amazing data visualization tool for Prometheus metrics. You can create your own custom dashboards and dive a lot deeper into each component while debugging.

Does it work with Thanos too?

Yes, in fact I've been developing this against my little Thanos cluster most of the time.
The queries even dynamically add headers for downsampling and disable partial responses.

How many instances should I deploy?

It depends on the topology of your infrastructure, however, we think that alerting should still happen within each individual Prometheus and therefore running one instance with one Prometheus (pair) makes the most sense. Pyrra itself only needs one instance per Prometheus (pair).

Why don't you support more complex SLOs?

For now, we try to accomplish an easy-to-setup workflow for the most common SLOs. It is still possible to write these more complex SLOs manually and deploy them to Prometheus along those generated. You can base more complex SLOs on the output of one SLO from this tool.

Why is the objective target a string not a float?

Kubebuilder doesn't support floats in CRDs...
Therefore, we need to pass it as string and internally convert it from string to float64.

Here are some related projects:

Rohlik/pyrra