[kube-prometheus-stack] err="parsing YAML file /etc/prometheus/config_out/prometheus.env.yaml: empty duration string"

[nikskiz]

Describe the bug a clear and concise description of what the bug is.

When installing the chart using default values with helm install -n prometheus prometheus prometheus-community/kube-prometheus-stack --version=57.0.3 --create-namespace the pod prometheus-prometheus-kube-prometheus-prometheus-0 container prometheus crashes with the error

ts=2024-06-13T01:25:02.752Z caller=main.go:512 level=error msg="Error loading config (--config.file=/etc/prometheus/config_out/prometheus.env.yaml)" file=/etc/prometheus/config_out/prometheus.env.yaml err="parsing YAML file /etc/prometheus/config_out/prometheus.env.yaml: empty duration string"

What's your helm version?

v3.11.1

What's your kubectl version?

v1.30.2

Which chart?

prometheus-community/kube-prometheus-stack

What's the chart version?

57.0.3

What happened?

I install the helm chart in a EKS cluster and I receive an error.

What you expected to happen?

For the helm release to be deployed successfully with default values.

How to reproduce it?

1. run command helm install -n prometheus prometheus prometheus-community/kube-prometheus-stack --version=57.0.3 --create-namespace
2. Oberserve the pod prometheus-prometheus-kube-prometheus-prometheus-0

Enter the changed values of values.yaml?

n/a

Enter the command that you execute and failing/misfunctioning.

helm install -n prometheus prometheus prometheus-community/kube-prometheus-stack --version=57.0.3 --create-namespace

Anything else we need to know?

No response

[ZanderPittari]

I'm getting the same thing

[zeritti]

I cannot reproduce the error with a fresh install of that same chart release.

I suspect that outdated/incompatible Prometheus operator's CRDs or remnants thereof are present in your cluster and as a result, some intervals do not get their default values set and remain empty, most likely prometheus' global.evaluation_interval and global.scrape_interval that do not get default values from the chart (corresponding values prometheus.prometheusSpec.scrapeInterval and prometheus.prometheusSpec.evaluationInterval are empty).

Please, make sure to upgrade the CRDs first from the same release as that of the operator (Prometheus operator v0.72.0 with chart release 57). More in README.

[ZanderPittari]

I managed to fix it with these lines added to prometheus.prometheusSpec:
scrapeInterval: 30s
evaluationInterval: 30s

e.g.

Found it from another issue on this GitHub. Forgot the issue number though

[nikskiz]

Thank you @zeritti I followed the instructions and managed to get it working 🙏

kubectl delete crd prometheusagents.monitoring.coreos.com
kubectl delete crd prometheuses.monitoring.coreos.com
kubectl delete crd prometheusrules.monitoring.coreos.com

Then I installed with:

helm install -n prometheus prometheus  prometheus-community/kube-prometheus-stack --version=57.0.3 --create-namespace