pulp/pulpcore

Replace built-in metrics by custom instrumentation middleware

Closed this issue · 2 comments

Recently, we discovered that vendoring a customized version of our instrumentation middleware offers significant advantages, primarily due to the added flexibility it provides for managing metric attributes. Upstream instrumentation libraries lack options for hooking into metrics to modify their behavior. These metrics often include attributes that do not apply to standard use cases. Additionally, contributing upstream seems to be a bit challenging, as spans and metrics are tightly coupled in their implementation.

The goal of this task is to:

  1. remove the opentelemetry-instrumentation-* dependencies and switch to the instrumentation-sdk only,
  2. introduce a custom middleware for api-app and content-app, optionally, for worker-app,
  3. remove the current testing machinery for traces and metrics and replace it with reasonable unit tests,
  4. introduce a new setting that activates the opentelemetry instrumentation on the application's side.

We already have a working concept in the pulp-service repository:
content-app: https://github.com/pulp/pulp-service/blob/4a72206d961b0d3e751405850b27bdc366414682/pulp_service/pulp_service/app/content.py#L69
api-app: https://github.com/pulp/pulp-service/blob/4a72206d961b0d3e751405850b27bdc366414682/pulp_service/pulp_service/app/middleware.py#L74

We are also planning to remove OTEL_PREFIX and PULP_OTEL_ENABLED from our container image:
image

We will remove OTEL_PREFIX entirely as we will no longer use it. The PULP_OTEL_ENABLED env var will be moved to be proper settings in pulp as an application and it will be called OTEL_ENABLED.