kubernetes/kops

Allow for setting `--disabled-metrics` within KubeletConfigSpec

Closed this issue · 2 comments

jandersen-plaid commented

/kind feature

1. Describe IN DETAIL the feature/behavior/change you would like to see.

Currently, there is no way to tune the metrics that individual kubelet's expose on their Prometheus compatible endpoint via kops. Instead, you need to drop the metrics that you do not want at the ServiceMonitor, PodMonitor, or Prometheus-compatible configuration level (via drop or relabel). This works, but causes some amount of increased processing on the observability layer because the metrics still need to be ingested. Additionally, this has cost consequences for the data sent over the wire, since the Prometheus-compatible tooling must scrape in the data that is never used.

Kubelet does offer a flag to support this, in the form of --disabled-metrics which takes a comma separated list of metrics that can be disabled. Ideally, this flag would be supported in the Kops KubeletConfigSpec so that the metrics can be disabled at will, and rolled out during a cluster update.

2. Feel free to provide a design supporting your feature request.

The design here is pretty simple:

  1. Add DisabledMetrics as a field in the KubeletConfigSpec which is a list of strings
  2. Those strings then get translated into --disabled-metrics=metric_name1,metric_name2,... when the kubelet command line parameters are generated.

Happy to contribute this as a feature since it is relatively straightforward!

jandersen-plaid commented

I take that back, apparently --disabled-metrics is not supported in kubelet at all. I will file an issue upstream and then come back around if it gets picked up.

jandersen-plaid commented

Closing this issue as the flag needs to be implemented upstream in kubelet or the documentation needs to be changed for --disabled-metrics neither of which fall under the responsibilities of kops.