prometheus-community/helm-charts

[kube-prometheus-stack] thanos-ruler can no longer communicate with Alertmanager

TomHellier opened this issue · 1 comments

Describe the bug a clear and concise description of what the bug is.

It appears that since the update of alertmanager to 0.27.0 by @DrFaust92, a change to alertmanager has deprecated the /v1/ api.

It looks like the default behaviour will be updated in the next release of thanos-rule.

Alerts generated by the thanos-ruler no longer get propagated to the alertmanager instance.

ts=2024-03-22T12:16:26.773542094Z caller=alert.go:326 level=warn component=rules msg="sending alerts failed" err="bad response status 410 Gone from \"http://monitoring-alertmanager.monitoring.svc.cluster.local:9093/alertmanager/api/v1/alerts\""

What's your helm version?

argocd v2.10.3

What's your kubectl version?

1.28

Which chart?

kube-prometheus-stack

What's the chart version?

57.0.3

What happened?

Alertmanager has deprecated the v1 api, and no longer accepts messages from thanos-ruler

What you expected to happen?

alerting continued to work

How to reproduce it?

Using the kube prometheus stack, ensure thanos-ruler is alerting, and observe the error logs in thanos-ruler, and an empty alertmanager

Enter the changed values of values.yaml?

NONE

Enter the command that you execute and failing/misfunctioning.

None

Anything else we need to know?

No response

Im not sure anything needs to be done on the chart side, you can either pin AM version to 0.26.0 or change AM config for API as referenced in the ticket