Generate prometheus metrics
Grounz opened this issue · 8 comments
Feature.
In production environment we want get alerts when a chart become deprecated, or when a chart is not up to date
Description
For each helm charts found on K8S cluster nova create a prometheus metrics. This metrics contains many labels.
The metrics nove_helm_uptodate_metrics will be equal to 1 if chart deployed is not up to date.
nova_helm_uptodate_chart{chart="cert-manager", installed="v0.11.0", latest=" v0.15.0", deprecated="false"} 1
And other metrics should be a metrics on a deprecated chart:
If chart is deprecated the metrics equal 1 otherwise equal 0.
nova_helm_deprecated_charts{chart="cert-manager", installed="v0.11.0", latest=" v0.15.0", deprecated="true"} 1
And after prometheus scrap nova and get metrics and we can create this alerts:
nova_helm_uptodate_chart < 0
nova_helm_deprecated_charts < 0
@Grounz thanks for the request!
Nova doesn't run as a long-lived process, so I'm not sure how to do this without adding a very large amount of code to turn Nova into a daemon of some kind. It's probably possible, but in my mind doesn't line up with the current workflow of Nova. Would it be better to maybe add prometheus push support and run it as a cronjob?
Another option would be to have a separate wrapper and prom exporter for it, but that seems a heavy lift too. @makoscafee @rbren curious what your thought are here since you're the codeowners :-D
Agreed - definitely a big lift to get something log lived in the cluster.
IMO it's probably out of scope for this project, but I'm open to it.
Maybe we leave this ticket open, and see if other folks feel strongly about this
Thanks to you, and thanks for this amazing tools :p
@rbren @sudermanjr Oh ok, if it's out of scope, Maybe use push gateway method it's possible and less complicated ?
Now, i don't know any prometheus exporter of helm chart status (version and deprecated) this idea seems to me very good for SRE teams. In big K8S cluster it's difficult to follow chart update state.
Maybe we can create nova_exporter which use nova go package and exposed metrics or push it to push gateway.
what do you think about that ?
I think a separate exporter could work, but also likely outside the scope of this repo. If someone wanted to create that separately I think it might work. I don't think we have tested the external usability of the Nova packages, so there might be issues with that. A wrapper that shells out could be functional as well.
I am with @rbren, I think we should leave this open and see if it gets additional votes or interest.
I want that! But I cannot reopen :/
There's also the possibility that nova creates ressources of reports (using a specific CRD) on the Kubernetes etcd, and then have a micro service retrieving the last CRD to expose it as prometheus metrics.
It might be simpler.
Hello guys,
we have created an exporter with this exact same purpose: https://github.com/LeRaymy/helm-deprecated-exporter
We use nova to create a config map with all the infos, and an exporter to parse all the infos
Really an alpha project, but working for our needs :) (Feel free to test and report us any issues! )
That's great! I'll give it a try!
I used nova as a CLI to do exactly that!