log.level setting is ignored
gfrankliu opened this issue · 8 comments
This version of prometheus seems to ignore log.level settings.
eg: Running prometheus-2.28.1-gmp.4 with below test config:
$ cat test.yaml
global:
scrape_interval: 2s
scrape_configs:
- job_name: 'test_job'
static_configs:
- targets: ['non-exist-host:9090']
$ ./prometheus --log.level=error --config.file=./test.yaml
ts=2022-03-10T05:27:52.806Z caller=log.go:124 component="scrape manager" scrape_pool=test_job level=debug target=http://non-exist-host:9090/metrics msg="Scrape failed" err="Get \"http://non-exist-host:9090/metrics\": dial tcp: lookup non-exist-host on 127.0.0.1:53: no such host"
ts=2022-03-10T05:27:54.600Z caller=log.go:124 component="scrape manager" scrape_pool=test_job level=debug target=http://non-exist-host:9090/metrics msg="Scrape failed" err="Get \"http://non-exist-host:9090/metrics\": dial tcp: lookup non-exist-host on 127.0.0.1:53: no such host"
...
<keeps repeating every scrape interval>
Even though --log.level=error is used when running prometheus, it still logs debug messages: ... scrape_pool=test_job level=debug ...
If I use the official prometheus-2.28.1:
$ ./prometheus --log.level=error --config.file=./test.yaml
I don't see any debug logs.
I was confused first because I couldn't reproduce this in general for various debug log lines.
Causing that particular log line to trigger however, has the same issue for me.
Turns out apparently in our code the import path got renamed from github.com/go-kit/log/level to github.com/go-kit/kit/log/level. Presmuably by some tool. So while the code is largely the same, this one isn't configured by the flags.
We'll fix this with the next release. Thanks for bringing this up.
Thanks and I do see bunch of matches:
$ grep -r go-kit/kit/log prometheus-2.28.1-gmp.4/
prometheus-2.28.1-gmp.4/scrape/scrape.go: "github.com/go-kit/kit/log"
prometheus-2.28.1-gmp.4/scrape/scrape.go: "github.com/go-kit/kit/log/level"
prometheus-2.28.1-gmp.4/vendor/github.com/go-kit/kit/log/README.md: kitlog "github.com/go-kit/kit/log"
prometheus-2.28.1-gmp.4/vendor/github.com/go-kit/kit/log/README.md:Log levels are supported via the [level package](https://godoc.org/github.com/go-kit/kit/log/level).
prometheus-2.28.1-gmp.4/vendor/github.com/go-kit/kit/log/README.md:like improvements to [the leveled logger](https://godoc.org/github.com/go-kit/kit/log/level),
prometheus-2.28.1-gmp.4/vendor/github.com/go-kit/kit/log/README.md:- Be composable with [contextual loggers](https://godoc.org/github.com/go-kit/kit/log#With),
prometheus-2.28.1-gmp.4/vendor/github.com/go-kit/kit/log/README.md:- Not break the behavior of [log.Caller](https://godoc.org/github.com/go-kit/kit/log#Caller) in any wrapped contextual loggers, and
prometheus-2.28.1-gmp.4/vendor/github.com/go-kit/kit/log/level/level.go:import "github.com/go-kit/kit/log"
prometheus-2.28.1-gmp.4/vendor/github.com/GoogleCloudPlatform/prometheus-engine/pkg/lease/lease.go: "github.com/go-kit/kit/log"
prometheus-2.28.1-gmp.4/vendor/github.com/GoogleCloudPlatform/prometheus-engine/pkg/export/setup/setup.go: "github.com/go-kit/kit/log"
prometheus-2.28.1-gmp.4/vendor/github.com/GoogleCloudPlatform/prometheus-engine/pkg/export/export.go: "github.com/go-kit/kit/log"
prometheus-2.28.1-gmp.4/vendor/github.com/GoogleCloudPlatform/prometheus-engine/pkg/export/export.go: "github.com/go-kit/kit/log/level"
prometheus-2.28.1-gmp.4/vendor/github.com/GoogleCloudPlatform/prometheus-engine/pkg/export/series_cache.go: "github.com/go-kit/kit/log"
prometheus-2.28.1-gmp.4/vendor/github.com/GoogleCloudPlatform/prometheus-engine/pkg/export/series_cache.go: "github.com/go-kit/kit/log/level"
prometheus-2.28.1-gmp.4/vendor/k8s.io/klog/README.md:in projects that use the [go-kit logger](https://godoc.org/github.com/go-kit/kit/log).
prometheus-2.28.1-gmp.4/vendor/k8s.io/klog/.errcheck_excludes:(github.com/go-kit/kit/log.Logger).Log
prometheus-2.28.1-gmp.4/vendor/k8s.io/klog/v2/klog.go: "github.com/go-kit/kit/log"
prometheus-2.28.1-gmp.4/vendor/k8s.io/klog/v2/klog.go: "github.com/go-kit/kit/log/level"
prometheus-2.28.1-gmp.4/vendor/k8s.io/klog/klog.go: "github.com/go-kit/kit/log"
prometheus-2.28.1-gmp.4/vendor/k8s.io/klog/klog.go: "github.com/go-kit/kit/log/level"
prometheus-2.28.1-gmp.4/vendor/modules.txt:github.com/go-kit/kit/log
prometheus-2.28.1-gmp.4/vendor/modules.txt:github.com/go-kit/kit/log/level
prometheus-2.28.1-gmp.4/cmd/prometheus/main.go: kitloglevel "github.com/go-kit/kit/log/level"
prometheus-2.28.1-gmp.4/.golangci.yml: # - github.com/go-kit/kit/log: "Use github.com/go-kit/log instead of github.com/go-kit/kit/log"
We fixed the occurrence you mentioned for the next release, which is probably the main one with spam potential. We will change the one in our prometheus-engine dependency for the release after that.
The other ones you see there are beyond our control and Prometheus appears to do some shimming to handle that case.
That works, thanks!
I downloaded prometheus-2.28.1-gmp.5 and the log.level is working now. The ./prometheus --version still shows 2.28.1-gmp.4 probably because VERSION file didn't get updated.
Even though I set --log.level=error, I still see "info" logs every 10 min, not spamming but it shows log.level setting is not honored by gcm_exporter:
Mar 12 06:39:05 rls-a prometheus[2079932]: ts=2022-03-12T06:39:05.700Z caller=log.go:124 component=gcm_exporter level=info msg="garbage collection completed" took=155.996µs seriesPurged=0
I tried
find . -type f | xargs sed -i 's,go-kit/kit/log,go-kit/log,g'`
and rebuilt it, now gcm_exporter component honors log.level
Thanks for digging into this. The adjustment of dependencies in our prometheus-engine repo did not make it into the recent release. We'll update there and then with the next release, this will apply to the Prometheus binary as well.