tidb dashboard 监控指标采集问题
soulmz opened this issue · 9 comments
soulmz commented
Bug Report
What version of Kubernetes are you using?
k8s v1.27.8
tidb-cluster 7.5.0
What version of TiDB Operator are you using?
TiDB Operator Version: version.Info{GitVersion:"v1.5.2", GitCommit:"456a0273f67ac61212da78956f49f0a4a07e21d8", GitTreeState:"clean", BuildDate:"2024-01-19T03:50:22Z", GoVersion:"go1.21.5", Compiler:"gc", Platform:"linux/amd64"}
What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?
openEBS localPV
What's the status of the TiDB cluster pods?
What did you do?
What did you expect to see?
dashboard 有关 tidb 组件监控数据存在问题。
这个指标有问题，rate(process_cpu_seconds_total{job="tidb"}[120s])
通过 prometheus 查看，发现 job 不匹配。
dashboard grafana url 链接地址错误。
部分监控指标不显示，初步排查，发现都是 promQL 表达式有问题。
What did you see instead?
soulmz commented
请问这个表达式是否是代码写死的？
如果不是写死的，能否通过数据库手动改正？
查阅官方文档，没发现修改的方式。
csuzhangxc commented
This is caused by the job_name mismatch of TidbMonitor (Prometheus) and Dashboard promQL.
You can try to use configMapRef of TidbMonitor to mount an external ConfigMap to overwrite the default prometheus.yml in ClusterName-monitor ConfigMap with Dashboard expected job_name (e.g tidb, not ns-clusterName-tidb).
ref https://github.com/pingcap/tidb-operator/blob/v1.5.2/examples/advanced/tidb-monitor.yaml#L192-L197
soulmz commented
This is caused by the
job_namemismatch of TidbMonitor (Prometheus) and Dashboard promQL.You can try to use
configMapRefof TidbMonitor to mount an external ConfigMap to overwrite the defaultprometheus.ymlinClusterName-monitorConfigMap with Dashboard expectedjob_name(e.gtidb, notns-clusterName-tidb).ref https://github.com/pingcap/tidb-operator/blob/v1.5.2/examples/advanced/tidb-monitor.yaml#L192-L197
tidb-dashboard 这个是写死了在 ts 代码里吗？我使用 外部的configmap 方式没找到具体的覆盖 tidb-dashboard 面板上的监控指标。
csuzhangxc commented
This is caused by the
job_namemismatch of TidbMonitor (Prometheus) and Dashboard promQL.
You can try to useconfigMapRefof TidbMonitor to mount an external ConfigMap to overwrite the defaultprometheus.ymlinClusterName-monitorConfigMap with Dashboard expectedjob_name(e.gtidb, notns-clusterName-tidb).
ref https://github.com/pingcap/tidb-operator/blob/v1.5.2/examples/advanced/tidb-monitor.yaml#L192-L197
also ref https://github.com/pingcap/tidb-dashboard/blob/b8552047eebc8d34e911cf1e1505347599482ac1/ui/packages/tidb-dashboard-for-op/src/apps/Monitoring/metricsQueries.ts#L612tidb-dashboard 这个是写死了在 ts 代码里吗？我使用 外部的
configmap方式没找到具体的覆盖 tidb-dashboard 面板上的监控指标。
I mean, use an external ConfigMap to overwite the metrics labels (job_name) in Prometheus so that they can match the hardcoded PromQL in tidb-dashboard.
soulmz commented
This is caused by the
job_namemismatch of TidbMonitor (Prometheus) and Dashboard promQL.
You can try to useconfigMapRefof TidbMonitor to mount an external ConfigMap to overwrite the defaultprometheus.ymlinClusterName-monitorConfigMap with Dashboard expectedjob_name(e.gtidb, notns-clusterName-tidb).
ref https://github.com/pingcap/tidb-operator/blob/v1.5.2/examples/advanced/tidb-monitor.yaml#L192-L197
also ref https://github.com/pingcap/tidb-dashboard/blob/b8552047eebc8d34e911cf1e1505347599482ac1/ui/packages/tidb-dashboard-for-op/src/apps/Monitoring/metricsQueries.ts#L612tidb-dashboard 这个是写死了在 ts 代码里吗？我使用 外部的
configmap方式没找到具体的覆盖 tidb-dashboard 面板上的监控指标。I mean, use an external ConfigMap to overwite the metrics labels (job_name) in Prometheus so that they can match the hardcoded PromQL in tidb-dashboard.
我通过 修改了旧的 prometheus ，增加了 job_name: tidb
global:
evaluation_interval: 15s
scrape_interval: 15s
external_labels:
prometheus_replica: tidb-cluster_ex-monitor-0
scrape_configs:
- job_name: tidb-cluster-ex-pd
honor_labels: true
scrape_interval: 15s
scheme: http
kubernetes_sd_configs:
- api_server: null
role: pod
namespaces:
names:
- tidb-cluster
tls_config:
insecure_skip_verify: true
relabel_configs:
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
action: keep
regex: ex
- source_labels:
- __meta_kubernetes_namespace
action: keep
regex: tidb-cluster
- source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scrape
action: keep
regex: "true"
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_component
action: keep
regex: pd
- action: replace
regex: (.+);(.+);(.+);(.+)
replacement: $1.$2-pd-peer.$3:$4
target_label: __address__
source_labels:
- __meta_kubernetes_pod_name
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_annotation_prometheus_io_port
- source_labels:
- __meta_kubernetes_namespace
action: replace
target_label: kubernetes_namespace
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
action: replace
target_label: cluster
- source_labels:
- __meta_kubernetes_pod_name
action: replace
target_label: instance
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_component
action: replace
target_label: component
- source_labels:
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
separator: '-'
target_label: tidb_cluster
- source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_path
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels:
- __address__
action: hashmod
target_label: __tmp_hash
modulus: 1
- source_labels:
- __tmp_hash
regex: 0
action: keep
- job_name: tidb-cluster-ex-tidb
honor_labels: true
scrape_interval: 15s
scheme: http
kubernetes_sd_configs:
- api_server: null
role: pod
namespaces:
names:
- tidb-cluster
tls_config:
insecure_skip_verify: true
relabel_configs:
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
action: keep
regex: ex
- source_labels:
- __meta_kubernetes_namespace
action: keep
regex: tidb-cluster
- source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scrape
action: keep
regex: "true"
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_component
action: keep
regex: tidb
- action: replace
regex: (.+);(.+);(.+);(.+)
replacement: $1.$2-tidb-peer.$3:$4
target_label: __address__
source_labels:
- __meta_kubernetes_pod_name
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_annotation_prometheus_io_port
- source_labels:
- __meta_kubernetes_namespace
action: replace
target_label: kubernetes_namespace
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
action: replace
target_label: cluster
- source_labels:
- __meta_kubernetes_pod_name
action: replace
target_label: instance
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_component
action: replace
target_label: component
- source_labels:
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
separator: '-'
target_label: tidb_cluster
- source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_path
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels:
- __address__
action: hashmod
target_label: __tmp_hash
modulus: 1
- source_labels:
- __tmp_hash
regex: 0
action: keep
# add tidb
- job_name: tidb
honor_labels: true
scrape_interval: 15s
scheme: http
kubernetes_sd_configs:
- api_server: null
role: pod
namespaces:
names:
- tidb-cluster
tls_config:
insecure_skip_verify: true
relabel_configs:
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
action: keep
regex: ex
- source_labels:
- __meta_kubernetes_namespace
action: keep
regex: tidb-cluster
- source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scrape
action: keep
regex: "true"
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_component
action: keep
regex: tidb
- action: replace
regex: (.+);(.+);(.+);(.+)
replacement: $1.$2-tidb-peer.$3:$4
target_label: __address__
source_labels:
- __meta_kubernetes_pod_name
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_annotation_prometheus_io_port
- source_labels:
- __meta_kubernetes_namespace
action: replace
target_label: kubernetes_namespace
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
action: replace
target_label: cluster
- source_labels:
- __meta_kubernetes_pod_name
action: replace
target_label: instance
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_component
action: replace
target_label: component
- source_labels:
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
separator: '-'
target_label: tidb_cluster
- source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_path
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels:
- __address__
action: hashmod
target_label: __tmp_hash
modulus: 1
- source_labels:
- __tmp_hash
regex: 0
action: keep
- job_name: tidb-cluster-ex-tikv
honor_labels: true
scrape_interval: 15s
scheme: http
kubernetes_sd_configs:
- api_server: null
role: pod
namespaces:
names:
- tidb-cluster
tls_config:
insecure_skip_verify: true
relabel_configs:
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
action: keep
regex: ex
- source_labels:
- __meta_kubernetes_namespace
action: keep
regex: tidb-cluster
- source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scrape
action: keep
regex: "true"
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_component
action: keep
regex: tikv
- action: replace
regex: (.+);(.+);(.+);(.+)
replacement: $1.$2-tikv-peer.$3:$4
target_label: __address__
source_labels:
- __meta_kubernetes_pod_name
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_annotation_prometheus_io_port
- source_labels:
- __meta_kubernetes_namespace
action: replace
target_label: kubernetes_namespace
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
action: replace
target_label: cluster
- source_labels:
- __meta_kubernetes_pod_name
action: replace
target_label: instance
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_component
action: replace
target_label: component
- source_labels:
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
separator: '-'
target_label: tidb_cluster
- source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_path
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels:
- __address__
action: hashmod
target_label: __tmp_hash
modulus: 1
- source_labels:
- __tmp_hash
regex: 0
action: keep
- job_name: tidb-cluster-ex-tiproxy
honor_labels: true
scrape_interval: 15s
scheme: http
kubernetes_sd_configs:
- api_server: null
role: pod
namespaces:
names:
- tidb-cluster
tls_config:
insecure_skip_verify: true
relabel_configs:
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
action: keep
regex: ex
- source_labels:
- __meta_kubernetes_namespace
action: keep
regex: tidb-cluster
- source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scrape
action: keep
regex: "true"
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_component
action: keep
regex: tiproxy
- action: replace
regex: (.+);(.+);(.+);(.+)
replacement: $1.$2-tiproxy-peer.$3:$4
target_label: __address__
source_labels:
- __meta_kubernetes_pod_name
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_annotation_prometheus_io_port
- source_labels:
- __meta_kubernetes_namespace
action: replace
target_label: kubernetes_namespace
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
action: replace
target_label: cluster
- source_labels:
- __meta_kubernetes_pod_name
action: replace
target_label: instance
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_component
action: replace
target_label: component
- source_labels:
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
separator: '-'
target_label: tidb_cluster
- source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_path
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels:
- __address__
action: hashmod
target_label: __tmp_hash
modulus: 1
- source_labels:
- __tmp_hash
regex: 0
action: keep
- job_name: tidb-cluster-ex-tiflash
honor_labels: true
scrape_interval: 15s
scheme: http
kubernetes_sd_configs:
- api_server: null
role: pod
namespaces:
names:
- tidb-cluster
tls_config:
insecure_skip_verify: true
relabel_configs:
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
action: keep
regex: ex
- source_labels:
- __meta_kubernetes_namespace
action: keep
regex: tidb-cluster
- source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scrape
action: keep
regex: "true"
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_component
action: keep
regex: tiflash
- action: replace
regex: (.+);(.+);(.+);(.+)
replacement: $1.$2-tiflash-peer.$3:$4
target_label: __address__
source_labels:
- __meta_kubernetes_pod_name
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_annotation_prometheus_io_port
- source_labels:
- __meta_kubernetes_namespace
action: replace
target_label: kubernetes_namespace
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
action: replace
target_label: cluster
- source_labels:
- __meta_kubernetes_pod_name
action: replace
target_label: instance
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_component
action: replace
target_label: component
- source_labels:
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
separator: '-'
target_label: tidb_cluster
- source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_path
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels:
- __address__
action: hashmod
target_label: __tmp_hash
modulus: 1
- source_labels:
- __tmp_hash
regex: 0
action: keep
- job_name: tidb-cluster-ex-tiflash-proxy
honor_labels: true
scrape_interval: 15s
scheme: http
kubernetes_sd_configs:
- api_server: null
role: pod
namespaces:
names:
- tidb-cluster
tls_config:
insecure_skip_verify: true
relabel_configs:
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
action: keep
regex: ex
- source_labels:
- __meta_kubernetes_namespace
action: keep
regex: tidb-cluster
- source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scrape
action: keep
regex: "true"
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_component
action: keep
regex: tiflash
- action: replace
regex: (.+);(.+);(.+);(.+)
replacement: $1.$2-tiflash-peer.$3:$4
target_label: __address__
source_labels:
- __meta_kubernetes_pod_name
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_annotation_tiflash_proxy_prometheus_io_port
- source_labels:
- __meta_kubernetes_namespace
action: replace
target_label: kubernetes_namespace
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
action: replace
target_label: cluster
- source_labels:
- __meta_kubernetes_pod_name
action: replace
target_label: instance
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_component
action: replace
target_label: component
- source_labels:
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
separator: '-'
target_label: tidb_cluster
- source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_path
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels:
- __address__
action: hashmod
target_label: __tmp_hash
modulus: 1
- source_labels:
- __tmp_hash
regex: 0
action: keep
- job_name: tidb-cluster-ex-pump
honor_labels: true
scrape_interval: 15s
scheme: http
kubernetes_sd_configs:
- api_server: null
role: pod
namespaces:
names:
- tidb-cluster
tls_config:
insecure_skip_verify: true
relabel_configs:
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
action: keep
regex: ex
- source_labels:
- __meta_kubernetes_namespace
action: keep
regex: tidb-cluster
- source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scrape
action: keep
regex: "true"
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_component
action: keep
regex: pump
- action: replace
regex: (.+);(.+);(.+);(.+)
replacement: $1.$2-pump.$3:$4
target_label: __address__
source_labels:
- __meta_kubernetes_pod_name
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_annotation_prometheus_io_port
- source_labels:
- __meta_kubernetes_namespace
action: replace
target_label: kubernetes_namespace
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
action: replace
target_label: cluster
- source_labels:
- __meta_kubernetes_pod_name
action: replace
target_label: instance
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_component
action: replace
target_label: component
- source_labels:
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
separator: '-'
target_label: tidb_cluster
- source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_path
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels:
- __address__
action: hashmod
target_label: __tmp_hash
modulus: 1
- source_labels:
- __tmp_hash
regex: 0
action: keep
- job_name: tidb-cluster-ex-drainer
honor_labels: true
scrape_interval: 15s
scheme: http
kubernetes_sd_configs:
- api_server: null
role: pod
namespaces:
names:
- tidb-cluster
tls_config:
insecure_skip_verify: true
relabel_configs:
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
action: keep
regex: ex
- source_labels:
- __meta_kubernetes_namespace
action: keep
regex: tidb-cluster
- source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scrape
action: keep
regex: "true"
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_component
action: keep
regex: drainer
- action: replace
regex: (.+);(.+);(.+);(.+)
replacement: $1.$2.$3:$4
target_label: __address__
source_labels:
- __meta_kubernetes_pod_name
- __meta_kubernetes_pod_label_app_kubernetes_io_name
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_annotation_prometheus_io_port
- source_labels:
- __meta_kubernetes_namespace
action: replace
target_label: kubernetes_namespace
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
action: replace
target_label: cluster
- source_labels:
- __meta_kubernetes_pod_name
action: replace
target_label: instance
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_component
action: replace
target_label: component
- source_labels:
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
separator: '-'
target_label: tidb_cluster
- source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_path
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels:
- __address__
action: hashmod
target_label: __tmp_hash
modulus: 1
- source_labels:
- __tmp_hash
regex: 0
action: keep
- job_name: tidb-cluster-ex-ticdc
honor_labels: true
scrape_interval: 15s
scheme: http
kubernetes_sd_configs:
- api_server: null
role: pod
namespaces:
names:
- tidb-cluster
tls_config:
insecure_skip_verify: true
relabel_configs:
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
action: keep
regex: ex
- source_labels:
- __meta_kubernetes_namespace
action: keep
regex: tidb-cluster
- source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scrape
action: keep
regex: "true"
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_component
action: keep
regex: ticdc
- action: replace
regex: (.+);(.+);(.+);(.+)
replacement: $1.$2-ticdc-peer.$3:$4
target_label: __address__
source_labels:
- __meta_kubernetes_pod_name
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_annotation_prometheus_io_port
- source_labels:
- __meta_kubernetes_namespace
action: replace
target_label: kubernetes_namespace
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
action: replace
target_label: cluster
- source_labels:
- __meta_kubernetes_pod_name
action: replace
target_label: instance
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_component
action: replace
target_label: component
- source_labels:
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
separator: '-'
target_label: tidb_cluster
- source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_path
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels:
- __address__
action: hashmod
target_label: __tmp_hash
modulus: 1
- source_labels:
- __tmp_hash
regex: 0
action: keep
- job_name: tidb-cluster-ex-lightning
honor_labels: true
scrape_interval: 15s
scheme: http
kubernetes_sd_configs:
- api_server: null
role: pod
namespaces:
names:
- tidb-cluster
tls_config:
insecure_skip_verify: true
relabel_configs:
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
action: keep
regex: ex
- source_labels:
- __meta_kubernetes_namespace
action: keep
regex: tidb-cluster
- source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scrape
action: keep
regex: "true"
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_component
action: keep
regex: tidb-lightning
- action: replace
regex: (.+);(.+);(.+);(.+)
replacement: $2.$3:$4
target_label: __address__
source_labels:
- __meta_kubernetes_pod_name
- __meta_kubernetes_pod_label_app_kubernetes_io_name
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_annotation_prometheus_io_port
- source_labels:
- __meta_kubernetes_namespace
action: replace
target_label: kubernetes_namespace
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
action: replace
target_label: cluster
- source_labels:
- __meta_kubernetes_pod_name
action: replace
target_label: instance
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_component
action: replace
target_label: component
- source_labels:
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
separator: '-'
target_label: tidb_cluster
- source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_path
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels:
- __address__
action: hashmod
target_label: __tmp_hash
modulus: 1
- source_labels:
- __tmp_hash
regex: 0
action: keep
rule_files:
- '/prometheus-rules/rules/*.rules.yml'
ok, 现在已经有数据了。但是部分指标名称已经更换了
例如
QPS 数据 tidb_executor_statement_total -> tidb_executor_statement_db_total
现在可以展示 tidb 部分数据了。指标命名更换了这个有办法吗？
csuzhangxc commented
QPS 数据 tidb_executor_statement_total -> tidb_executor_statement_db_total
What's the metrics name of getting from TiDB Pod's /metrics path directly?
soulmz commented
QPS 数据 tidb_executor_statement_total -> tidb_executor_statement_db_total
What's the metrics name of getting from TiDB Pod's
/metricspath directly?
dashboard 写的是 tidb_executor_statement_total 。
但是，tidb metrics 指标已经更换了 tidb_executor_statement_db_total 。
csuzhangxc commented
could you open an issue in https://github.com/pingcap/tidb
csuzhangxc commented
@soulmz Both of them should exist, https://github.com/pingcap/tidb/blob/v7.5.0/pkg/metrics/executor.go#L55-L69