KubeCon EU 2023 OpenTelemetry Kubernetes tutorial
Abstract: Deploying an end-to-end observability system comes with many challenges. The organization has to decide how data will be collected, what data formats will be used, sampling strategies, filter sensitive data (a.k.a. PII), and ultimately send data to the observability platform of their choice. In this session, we will teach you how to roll out end-to-end observability data collection on Kubernetes using the OpenTelemetry project. You will learn how to effectively instrument applications using auto-instrumentation, deploy the OpenTelemetry collector, and collect traces, metrics, and logs. You will gain the knowledge needed to tackle the mentioned challenges. After this session, you will be able to understand and use OpenTelemetry instrumentation libraries, collector and Kubernetes operator.
Schedule: https://kccnceu2023.sched.com/event/1HyZ3/
Slides: https://docs.google.com/presentation/d/1oDpQo9KW_C5HznE0GR53P22HzP_SN2Ml/edit#slide=id.g227fe4440bd_0_0 or slides.pdf
Prerequisites
This tutorial requires just requieres two major things:
- A tool like docker or podman to run OCI Container Images
- Access to a Kubernetes cluster within the version 1.19-1.26
In case you do not have access to a K8S cluster, you can use Kind or Minikube for a local Kubernetes cluster installations.
Here you will find our cluster setup.
Initial Setup
If you are using New Relic backend, create a secret
# Set a Kubernetes secret with your New Relic license key:
kubectl create secret generic newrelic-key-secret --from-literal=new_relic_license_key='<NEW_RELIC_LICENSE_KEY>'
The above secret is referenced in the ./backend/02-collector.yaml
Deploy cert-manager
cert-manager is used by OpenTelemetry operator to provision TLS certificates for admission webhooks.
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.yaml
Deploy observability backend
This course is all about Observabilty, so a backend is needed. If you don't have one, you can install a Prometheus compatible database Grafana Mimir as well as Loki for logs and Tempo for traces as follows:
kubectl apply -f https://raw.githubusercontent.com/pavolloffay/kubecon-eu-2023-opentelemetry-kubernetes-tutorial/main/backend/01-backend.yaml
Afterwards, the backend can be found in the namespace x. A grafana dashboard with preconfigured datasources and dashboard is available for visualisation and can be made accessible with port-fowarding.
kubectl port-forward -n observability-backend svc/grafana 3000:3000
Open it in the browser localhost:3000
Welcome to the OpenTelemetry Kubernetes tutorial.
Each tutorial step is located in a separate file:
- OpenTelemetry Collector introduction (Bene, 10 min)
- OpenTelemetry Operator introduction (Pavol, 10 min)
- Application instrumentation (Severin, Pavol, 30 min)
- Collecting metrics (Kristina, 15 min)
- Collecting logs (Yuri, 10 min)
- Roadmap (Yuri, 5 min)