In prometheus, all brokers seem to appear with the same data in each field, except for broker_id

[joan-s-molas]

The following fields appear to be the same for each one of the monitored nodes: instance, controller_revision_hash, kafka_broker_rack, kafka_broker_id, kubernetes_pod_name...

broker_id seems fine, however.

[joan-s-molas]

✔kafka_server_controller_channel_metrics_network_io_total{app="kafka",broker_id="2",controller_revision_hash="kafka-865867b745",instance="100.96.155.243:5556",job="kubernetes-pods",kafka_broker_id="0",kafka_broker_rack="eu-west-1a",kubernetes_namespace="darwin-dev",kubernetes_pod_name="kafka-0",statefulset_kubernetes_io_pod_name="kafka-0"}
✔kafka_server_controller_channel_metrics_network_io_total{app="kafka",broker_id="1",controller_revision_hash="kafka-865867b745",instance="100.96.155.243:5556",job="kubernetes-pods",kafka_broker_id="0",kafka_broker_rack="eu-west-1a",kubernetes_namespace="darwin-dev",kubernetes_pod_name="kafka-0",statefulset_kubernetes_io_pod_name="kafka-0"}
✔kafka_server_controller_channel_metrics_network_io_total{app="kafka",broker_id="0",controller_revision_hash="kafka-865867b745",instance="100.96.155.243:5556",job="kubernetes-pods",kafka_broker_id="0",kafka_broker_rack="eu-west-1a",kubernetes_namespace="darwin-dev",kubernetes_pod_name="kafka-0",statefulset_kubernetes_io_pod_name="kafka-0"}

[solsson]

I can confirm this, as surprised as you are :) In my case getting kafka_broker_id="1" for all three values.

I don't know why. Could "controller" have anything to do with it? Do brokers report each others' metric somehow?

@yacut do you know?

[joan-s-molas]

It doesn't seem to happen anymore. Maybe it had something wrong...