Warning: ADOT Collector v0.35.0 breaking changes - Normalization of metrics in prometheus exporters
rapphil opened this issue · 4 comments
What is happening?
Starting with version v0.85.0 of the upstream OpenTelemetry collector, the normalization of metrics when converting from OpenTelemetry metrics to Prometheus will be enabled by default.
When will this happen?
The AWS Distro for OpenTelemetry (ADOT) collector will start adopting the upstream behavior for name normalization of Prometheus metrics on version v0.35.0.
Who is affected and what to expect?
You are affected if you are using one of the following components present in the ADOT collector: prometheus exporter and prometheus remote write exporter.
If you don’t do anything, the name of the metrics exported by the collector using these components will be normalized according to well defined rules [1,2]. These rules are applied so best practices are applied to the naming of metrics [3].
Notably the following changes will enter in effect:
- The unit of the metrics will be appended as suffix to the metrics exported by Prometheus.
- Counters will have a
_total
suffix.
Examples:
Type | Name | Unit | Prometheus Metric exported |
---|---|---|---|
Gauge | system.filesystem.usage | By | system_filesystem_usage_bytes |
Gauge | system.network.dropped | packets | system_network_dropped_packets |
Counter | system.network.dropped | packets | system_network_dropped_packets_total |
Gauge | system.cpu.utilization | 1 | system_cpu_utilization_ratio |
The feature gate pkg.translator.prometheus.NormalizeName
is used to control the normalization behavior. If the feature gate is disabled, there won’t be name normalization. Besides that, both components added a configuration option to control the normalization behaviour in the component level, the add_metric_suffixes
property. This configuration option is only honoured if the feature gate is enabled.
In the ADOT collector v0.34.0, the feature gate pkg.translator.prometheus.NormalizeName
is disabled by default. In the collector v0.35.0, this feature gate will be enabled by default.
- [1] https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/translator/prometheus
- [2] https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/compatibility/prometheus_and_openmetrics.md#metric-metadata-1
- [3] https://prometheus.io/docs/practices/naming/
What action items do I need to take?
If you want to upgrade to the ADOT collector v0.35.0 without name normalization
This path is recommended for users who want to retain the metrics names unchanged after upgrading to the ADOT collector v0.35.0. This will guarantee that dashboards and alarms remain functional after the upgrade.
The following step by step guide is provided to guarantee a safe and gradual migration.
Steps:
- Upgrade the ADOT collector to v0.34.0.
- Modify/review the collector configuration for the components Prometheus remote write exporter and Prometheus exporter. They must add the property
add_metric_suffixes: false
to the configuration section of these components.
Example:
prometheusremotewrite:
endpoint: http://example.com
add_metric_suffixes: false # <-- Disable normalization
resource_to_telemetry_conversion:
enabled: true
auth:
authenticator: sigv4auth
exporters:
prometheus:
endpoint: "1.2.3.4:1234"
namespace: test-space
const_labels:
label1: value1
"another label": spaced value
send_timestamps: true
metric_expiration: 180m
enable_open_metrics: true
add_metric_suffixes: false # <-- Disable normalization
resource_to_telemetry_conversion:
enabled: true
- Upgrade the ADOT collector to v0.35.0.
If you recently upgraded to ADOT collector v0.35.0
This path is recommended for users that are already using the ADOT collector v0.35.0. The following steps are recommended:
- If you want to keep metric names as collected by the ADOT collector, e.g. untransformed, you should modify/review the collector configuration for the components Prometheus remote write exporter and Prometheus exporter. You must have the property
add_metric_suffixes: false
in the configuration section of these components.
Example:
prometheusremotewrite:
endpoint: http://example.com
add_metric_suffixes: false # <-- Disable normalization
resource_to_telemetry_conversion:
enabled: true
auth:
authenticator: sigv4auth
exporters:
prometheus:
endpoint: "1.2.3.4:1234"
namespace: test-space
const_labels:
label1: value1
"another label": spaced value
send_timestamps: true
metric_expiration: 180m
enable_open_metrics: true
add_metric_suffixes: false # <-- Disable normalization
resource_to_telemetry_conversion:
enabled: true
- If you want to use the name normalization of Prometheus metrics, you don’t need to do anything. We highly recommend that you verify that queries reference the normalized metric names.
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.
This issue was closed because it has been marked as stale for 30 days with no activity.