prometheus-community/helm-charts

n

aarifkhan8 opened this issue · 0 comments

aarifkhan8 commented

Hi Team,

I deployed prometheus with the help of helm in EKS:
I am getting below error in prometheus server:

kubectl logs prometheus-deployment-server-5c456d4b84-j9p9t -c prometheus-server -n prometheus

ts=2024-05-24T15:21:27.048Z caller=tls_config.go:277 level=info component=web msg="TLS is disabled." http2=false address=[::]:9090
ts=2024-05-24T15:21:27.048Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1704693600200 maxt=1704888000000 ulid=01HKT52HDZDFFXAQ2XSY7VPND8
.....
ts=2024-05-24T15:21:27.105Z caller=main.go:929 level=info msg="Stopping scrape manager..."
ts=2024-05-24T15:21:27.105Z caller=notifier.go:603 level=info component=notifier msg="Stopping notification manager..."
ts=2024-05-24T15:21:27.105Z caller=main.go:1150 level=info msg="Notifier manager stopped"
ts=2024-05-24T15:21:27.105Z caller=main.go:888 level=info msg="Notify discovery manager stopped"
ts=2024-05-24T15:21:27.105Z caller=main.go:921 level=info msg="Scrape manager stopped"
ts=2024-05-24T15:21:27.105Z caller=main.go:874 level=info msg="Scrape discovery manager stopped"
ts=2024-05-24T15:21:27.105Z caller=manager.go:995 level=info component="rule manager" msg="Starting rule manager..."
ts=2024-05-24T15:21:27.105Z caller=main.go:1159 level=error err="opening storage failed: reloadBlocks: 2 errors: corrupted block 01HJ60GPSBFAYK5VSFRF8VG839: mmap files: mmap, size 0: invalid argument; corrupted block 01HJZ462R17X9Z7ACTKTDSJ5Q7: mmap files: mmap, size 0: invalid argument"

In prometheus deployment pod:
level=error ts=2024-05-24T02:46:08.970009334Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post "http://127.0.0.1:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2024-05-24T02:46:13.970869157Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post "http://127.0.0.1:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2024-05-24T02:46:18.970271443Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post "http://127.0.0.1:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2024-05-24T02:46:23.970793336Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post "http://127.0.0.1:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"