Meerkat

Observability Starter Kit for JVM Applications

Introduction
Architecture overview
Quickstart
Why observability?
Components
Copyright

Introduction

Meerkat is a ready-to-deploy set of services for JVM applications utilizing OpenTelemetry, giving you a fully configured observability starting kit with logging, tracing, and metrics in a Kubernetes cluster. Install the components in your Kubernetes cluster, or first, give it a try in a localhost try-me environment with KinD (Kubernetes in Docker).

This starter kit is intended for developers and DevOps professionals who wish to implement observability in their JVM-based applications using Kubernetes.

Learn more about Meerkat in a dedicated blog series on SoftwareMill blog.

Architecture overview

With just a few simple commands install Meerkat components in a Kubernetes cluster:

OpenTelemetry Operator,
Grafana,
data backends that process telemetry data (logs, traces, and metrics),
a demo application - Spring Petclinic
MinIO buckets which are the object storage for data backends

OpenTelemetry Collector, managed by OpenTelemetry Operator, receives telemetry data from the instrumented demo application and forwards it to data backends: logs to Loki, traces to Tempo and metrics to Mimir. OpenTelemetry Operator injects auto-instrumentation into the demo app, simplifying the process by automatically collecting and sending telemetry data without manual code modifications. Java Instrumentation captures telemetry data from popular libraries and frameworks. Automatic instrumentation is a great way to start collecting standard metrics. To get specific metrics for your application, implement manual instrumentation. Once the data reaches Loki, Tempo, and Mimir, Grafana is configured to query these data sources directly. Grafana then visualizes the data in its dashboards, allowing you to explore and analyze the telemetry data.

If you only need the OpenTelemetry configuration and already have other backends installed (other than Loki, Mimir, or Tempo) in the Kubernetes cluster, you can use Meerkat partially. See the details here.

Quickstart

For more detailed instructions navigate to articles on the SoftwareMill blog:

Prerequisites

Docker installed - standalone Docker or Docker Desktop both work well
Pulumi CLI
Node.js and npm

Installation

Clone the Git repository:

git clone https://github.com/softwaremill/meerkat.git

Navigate to the meerkat folder:

cd meerkat

You can install the setup in your existing Kubernetes cluster or a test cluster managed by KinD on your localhost environment. If you already have a Kubernetes cluster, move to the Deploy the components section and skip the KinD cluster creation.

Create KinD cluster

First, you need to create a Kubernetes cluster on your localhost. For that, we're running a KinD cluster. If required, adjust the Kind configuration by modifying the try-me/kind/kind-config.yaml file. Run the command to install the cluster:

try-me/kind/cluster_create.sh

Deploy the components

The try-me/observability folder contains Pulumi code to deploy Observability components to the Kubernetes cluster. Components are deployed as Helm Charts. Make sure to connect to the correct Kubernetes context. By default, Pulumi will use a local kubeconfig if available. If you have just installed a Kind cluster, it should be your current context. Inside the try-me/observability folder install libraries. Run:

npm install

Initialize new Pulumi stack:

pulumi stack init localstack --no-select

Deploy necessary components:

pulumi up localstack

All components should be running after a few minutes. You can check if the demo app is running:

kubectl get pods -l app=petclinic

Patch the deployment with the annotation to start the automatic instrumentation:

kubectl patch deployment petclinic -n default -p '{"spec": {"template":{"metadata":{"annotations":{"instrumentation.opentelemetry.io/inject-java":"observability/jvm-autoinstrumentation"}}}} }'

You can use port-forwarding to access the PetClinic application UI and see how it looks and play with it:

kubectl port-forward services/petclinic 8888:8080

In your web browser enter http://localhost:8888/ and explore the demo app.

Retrieve password for Grafana:

kubectl get secret --namespace observability grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Use port-forwarding to access Grafana:

kubectl port-forward --namespace observability services/grafana 8000:80

Open your web browser and enter http://localhost:8000/. On the sign-in page, enter admin for the username and paste the password you retrieved from the secret. Explore dashboards and other features in Grafana.

Why observability?

With observability, you can gain full insight and visibility into your applications and infrastructure. It helps you pinpoint bottlenecks, identify the root causes of failures, optimize processes, and resolve problems in complex environments. Observability helps you manage and improve your systems, ensuring reliability and performance.

Components

We're using Kustomize to compose a list of manifests we install with one command. These manifests are required by the OpenTelemetry Operator and OpenTelemetry Collector. Kustomize is supported by many tools like kubectl, Terraform, Pulumi, FluxCD, ArgoCD, etc., which made us choose this tool.

Pulumi, besides installing data backends (more about this later), is also responsible for installing these required manifests in our installation.

We'll now briefly describe the components used in Meerkat.

Pulumi

Pulumi is an infrastructure-as-code tool. By using a general-purpose programming language you can declare what you want to deploy and apply this declaration by running the app.

Kustomize

Kustomize is a tool that lets you customize the configuration and simplifies its deployment. You can use it to create base templates and patches that modify only portions that differ between installations. In Meerkat, we use it to a very basic degree - to organize manifests into one kustomization and generate configMaps for Grafana dashboards.

Grafana

It's our open-source tool of choice for data visualization. Grafana uses telemetry data backends (Loki, Tempo, and Mimir) to create dashboards with graphs and tables. It also allows for easy data searching using LogQL, TraceQL, and PromQL.

Loki

Loki is a data backend for logs. It uses object storage for keeping data long-term. Any S3-compatible buckets can be used for that, in Meerkat we're using MinIO to keep everything hosted locally. Loki is being deployed using the simple scalable mode, which allows for scaling out individual pods if necessary.

Tempo

It's a backend for traces, it also stores data cheaply in object storage. Made by Grafana, it is easy to set up to work with Grafana, Loki, and Mimir. Tempo is being deployed using distributed mode, which allows for scaling every Tempo microservice separately.

Mimir

Same thing as Loki and Tempo, but for metrics. Mimir is being deployed in distributed mode.

OpenTelemetry Operator

The operator in the Kubernetes ecosystem is an extension that is responsible for the easy management of another application. It utilizes custom resources to describe how the managed application should work and how it should be configured. It's a way to automate processes that can be done by a Kubernetes cluster. We're using OpenTelemetry Operator to automate collector deployment and handle app instrumentation.

OpenTelemetry Collector

Everything is being powered by OpenTelemetry Collector. It's responsible for collecting, processing, and exporting the data to proper data backends. In Meerkat, we deploy it as a daemonset, which allows us to not only receive application data but also scrape nodes for resource utilization, etc.

softwaremill/meerkat