Multi-regional observability using Observatorium

Overview

This proof-of-concept (POC) deploys Observatorium components: API, Thanos, and Loki. It also deploys Prometheus, Grafana, and Promtail. See also architecture diagrams.

Projects used:

Deploying

You will need TLS certificates for Observatorium API, Prometheus, Grafana, and Promtail. See Creating TLS certificates for more details on how to generate them. Alternatively, Observatorium also supports OIDC (OpenID Connect) for authentication.

Create observatorium namespace. Everything else will be deployed into this namespace:

$ oc apply --kustomize observatorium-namespace/base

Observatorium requires an S3-compatible storage. If you don't have an existing S3 storage, you can leverage MinIO. Deploy MinIO S3 compatible object storage:

$ oc apply --kustomize minio/base

Install Observatorium operator:

$ oc apply --kustomize observatorium-operator/base

Deploy Observatorium. Search for REPLACE_ME string and customize manifests before deploying:

$ kustomize build observatorium-instance/overlays/logs | oc apply --filename -

Deploy Prometheus operator:

$ oc apply --kustomize prometheus-operator/base

Deploy Prometheus instance. Search for REPLACE_ME string and customize the manifests before deploying.

$ kustomize build prometheus-instance/base | oc apply --filename -

Deploy Promtail. Search for REPLACE_ME string and customize the manifests before deploying:

$ kustomize build promtail/overlays/observatorium | oc apply --filename -

Deploy Grafana operator:

$ oc apply --kustomize grafana-operator/base

Deploy Grafana instance. Search for REPLACE_ME string and customize the manifests before deploying:

$ oc apply --kustomize grafana-instance/base

Further tips

For testing purposes, you can expose some of the endpoints. Note that the commands below create OpenShift routes which expose unprotected services to the outside of the cluster. If this is an issue, use oc port-forward instead.

Create route for Prometheus UI:

$ oc expose svc prometheus-operated

Create route for Thanos Query UI:

$ oc expose svc observatorium-thanos-query --port http

Create route for Thanos Alerts Web UI (inactive, pending, firing alerts):

$ oc expose svc observatorium-thanos-rule --port http

Video

How to Collect and View OpenShift Cluster Logs Using Grafana Loki

Related documentation