Permission issues with prometheus/grafana
samlaf opened this issue · 0 comments
samlaf commented
Getting these issues when running docker compose up in tag 2.2.0 (same issues in 3.* and 4.*):
Attaching to cdn-up-and-running-backend-1, cdn-up-and-running-edge-1, cdn-up-and-running-nginx_base-1, monitoring_grafana, prometheus
prometheus | level=info ts=2023-02-11T20:48:25.178Z caller=main.go:333 msg="Starting Prometheus" version="(version=2.17.1, branch=HEAD, revision=ae041f97cfc6f43494bed65ec4ea4e3a0cf2ac69)"
prometheus | level=info ts=2023-02-11T20:48:25.178Z caller=main.go:334 build_context="(go=go1.13.9, user=root@806b02dfe114, date=20200326-16:18:19)"
prometheus | level=info ts=2023-02-11T20:48:25.178Z caller=main.go:335 host_details="(Linux 4.19.0-23-amd64 #1 SMP Debian 4.19.269-1 (2022-12-20) x86_64 f15eea78067e (none))"
prometheus | level=info ts=2023-02-11T20:48:25.178Z caller=main.go:336 fd_limits="(soft=1048576, hard=1048576)"
prometheus | level=info ts=2023-02-11T20:48:25.178Z caller=main.go:337 vm_limits="(soft=unlimited, hard=unlimited)"
prometheus | level=error ts=2023-02-11T20:48:25.179Z caller=query_logger.go:87 component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied"
prometheus | panic: Unable to create mmap-ed active query log
prometheus |
prometheus | goroutine 1 [running]:
prometheus | github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7ffdd6a4bed5, 0xb, 0x14, 0x2dcaa00, 0xc000641ef0, 0x2dcaa00)
prometheus | /app/promql/query_logger.go:117 +0x4cd
prometheus | main.main()
prometheus | /app/cmd/prometheus/main.go:364 +0x5245
monitoring_grafana | GF_PATHS_DATA='/var/lib/grafana' is not writable.
monitoring_grafana | You may have issues with file permissions, more information here: http://docs.grafana.org/installation/docker/#migrate-to-v51-or-later
monitoring_grafana | logger=settings t=2023-02-11T20:48:25.944018809Z level=info msg="Starting Grafana" version=9.3.6 commit=978237e7cb branch=HEAD compiled=2023-01-26T14:38:58Z
monitoring_grafana | logger=settings t=2023-02-11T20:48:25.944304168Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
monitoring_grafana | logger=settings t=2023-02-11T20:48:25.9443415Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
monitoring_grafana | logger=settings t=2023-02-11T20:48:25.944349525Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
monitoring_grafana | logger=settings t=2023-02-11T20:48:25.944354946Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
monitoring_grafana | logger=settings t=2023-02-11T20:48:25.94436247Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
monitoring_grafana | logger=settings t=2023-02-11T20:48:25.944369914Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
monitoring_grafana | logger=settings t=2023-02-11T20:48:25.944376087Z level=info msg="Config overridden from command line" arg="default.log.mode=console"
monitoring_grafana | logger=settings t=2023-02-11T20:48:25.944383461Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana"
monitoring_grafana | logger=settings t=2023-02-11T20:48:25.944410793Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana"
monitoring_grafana | logger=settings t=2023-02-11T20:48:25.944417186Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
monitoring_grafana | logger=settings t=2023-02-11T20:48:25.944437886Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
monitoring_grafana | logger=settings t=2023-02-11T20:48:25.944444298Z level=info msg="Config overridden from Environment variable" var="GF_SECURITY_ADMIN_USER=admin"
monitoring_grafana | logger=settings t=2023-02-11T20:48:25.944465589Z level=info msg="Config overridden from Environment variable" var="GF_SECURITY_ADMIN_PASSWORD=*********"
monitoring_grafana | logger=settings t=2023-02-11T20:48:25.94447149Z level=info msg="Config overridden from Environment variable" var="GF_USERS_ALLOW_SIGN_UP=false"
monitoring_grafana | logger=settings t=2023-02-11T20:48:25.944476841Z level=info msg="Path Home" path=/usr/share/grafana
monitoring_grafana | logger=settings t=2023-02-11T20:48:25.944482582Z level=info msg="Path Data" path=/var/lib/grafana
monitoring_grafana | logger=settings t=2023-02-11T20:48:25.944488332Z level=info msg="Path Logs" path=/var/log/grafana
monitoring_grafana | logger=settings t=2023-02-11T20:48:25.944493442Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
monitoring_grafana | logger=settings t=2023-02-11T20:48:25.944518621Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
monitoring_grafana | logger=settings t=2023-02-11T20:48:25.944524803Z level=info msg="App mode production"
monitoring_grafana | logger=sqlstore t=2023-02-11T20:48:25.944647669Z level=info msg="Connecting to DB" dbtype=sqlite3
monitoring_grafana | logger=sqlstore t=2023-02-11T20:48:25.944678529Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r-----
monitoring_grafana | logger=migrator t=2023-02-11T20:48:25.967090746Z level=info msg="Starting DB migrations"
monitoring_grafana | logger=migrator t=2023-02-11T20:48:25.971707872Z level=info msg="Executing migration" id="Update is_service_account column to nullable"
monitoring_grafana | logger=migrator t=2023-02-11T20:48:25.97185235Z level=error msg="Executing migration failed" id="Update is_service_account column to nullable" error="attempt to write a readonly database"
monitoring_grafana | logger=migrator t=2023-02-11T20:48:25.971874182Z level=error msg="Exec failed" error="attempt to write a readonly database" sql="ALTER TABLE user ADD COLUMN tmp_service_account BOOLEAN DEFAULT 0;\nUPDATE user SET tmp_service_account = is_service_account;\nALTER TABLE user DROP COLUMN is_service_account;\nALTER TABLE user RENAME COLUMN tmp_service_account TO is_service_account;"
monitoring_grafana | Failed to start grafana. error: migration failed (id = Update is_service_account column to nullable): attempt to write a readonly database
monitoring_grafana | migration failed (id = Update is_service_account column to nullable): attempt to write a readonly database
prometheus exited with code 0
prometheus | level=info ts=2023-02-11T20:48:26.475Z caller=main.go:333 msg="Starting Prometheus" version="(version=2.17.1, branch=HEAD, revision=ae041f97cfc6f43494bed65ec4ea4e3a0cf2ac69)"
prometheus | level=info ts=2023-02-11T20:48:26.475Z caller=main.go:334 build_context="(go=go1.13.9, user=root@806b02dfe114, date=20200326-16:18:19)"
prometheus | level=info ts=2023-02-11T20:48:26.475Z caller=main.go:335 host_details="(Linux 4.19.0-23-amd64 #1 SMP Debian 4.19.269-1 (2022-12-20) x86_64 f15eea78067e (none))"
prometheus | level=info ts=2023-02-11T20:48:26.475Z caller=main.go:336 fd_limits="(soft=1048576, hard=1048576)"
prometheus | level=info ts=2023-02-11T20:48:26.475Z caller=main.go:337 vm_limits="(soft=unlimited, hard=unlimited)"
prometheus | level=error ts=2023-02-11T20:48:26.477Z caller=query_logger.go:87 component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied"
prometheus | panic: Unable to create mmap-ed active query log
prometheus |
prometheus | goroutine 1 [running]:
prometheus | github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7ffc8a919ed5, 0xb, 0x14, 0x2dcaa00, 0xc000533c80, 0x2dcaa00)
prometheus | /app/promql/query_logger.go:117 +0x4cd
prometheus | main.main()
prometheus | /app/cmd/prometheus/main.go:364 +0x5245
monitoring_grafana exited with code 0
prometheus exited with code 0
monitoring_grafana | logger=settings t=2023-02-11T20:48:28.243144312Z level=info msg="Starting Grafana" version=9.3.6 commit=978237e7cb branch=HEAD compiled=2023-01-26T14:38:58Z
monitoring_grafana | logger=settings t=2023-02-11T20:48:28.243425683Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
monitoring_grafana | logger=settings t=2023-02-11T20:48:28.24344461Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
monitoring_grafana | logger=settings t=2023-02-11T20:48:28.243451092Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
monitoring_grafana | logger=settings t=2023-02-11T20:48:28.243456643Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
monitoring_grafana | logger=settings t=2023-02-11T20:48:28.243461903Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
monitoring_grafana | logger=settings t=2023-02-11T20:48:28.243852615Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
monitoring_grafana | logger=settings t=2023-02-11T20:48:28.243864027Z level=info msg="Config overridden from command line" arg="default.log.mode=console"
monitoring_grafana | logger=settings t=2023-02-11T20:48:28.24387099Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana"
monitoring_grafana | logger=settings t=2023-02-11T20:48:28.243885218Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana"
monitoring_grafana | logger=settings t=2023-02-11T20:48:28.24390689Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
monitoring_grafana | logger=settings t=2023-02-11T20:48:28.243918893Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
monitoring_grafana | logger=settings t=2023-02-11T20:48:28.243929613Z level=info msg="Config overridden from Environment variable" var="GF_SECURITY_ADMIN_USER=admin"
monitoring_grafana | logger=settings t=2023-02-11T20:48:28.243937688Z level=info msg="Config overridden from Environment variable" var="GF_SECURITY_ADMIN_PASSWORD=*********"
monitoring_grafana | logger=settings t=2023-02-11T20:48:28.243947798Z level=info msg="Config overridden from Environment variable" var="GF_USERS_ALLOW_SIGN_UP=false"
monitoring_grafana | logger=settings t=2023-02-11T20:48:28.243973798Z level=info msg="Path Home" path=/usr/share/grafana
monitoring_grafana | logger=settings t=2023-02-11T20:48:28.24399027Z level=info msg="Path Data" path=/var/lib/grafana
monitoring_grafana | logger=settings t=2023-02-11T20:48:28.244001552Z level=info msg="Path Logs" path=/var/log/grafana
monitoring_grafana | logger=settings t=2023-02-11T20:48:28.244015148Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
monitoring_grafana | logger=settings t=2023-02-11T20:48:28.244022031Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
monitoring_grafana | logger=settings t=2023-02-11T20:48:28.244033763Z level=info msg="App mode production"
monitoring_grafana | logger=sqlstore t=2023-02-11T20:48:28.244241694Z level=info msg="Connecting to DB" dbtype=sqlite3
monitoring_grafana | logger=sqlstore t=2023-02-11T20:48:28.24428607Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r-----
monitoring_grafana | logger=migrator t=2023-02-11T20:48:28.26636261Z level=info msg="Starting DB migrations"
monitoring_grafana | logger=migrator t=2023-02-11T20:48:28.270713545Z level=info msg="Executing migration" id="Update is_service_account column to nullable"
monitoring_grafana | logger=migrator t=2023-02-11T20:48:28.270845668Z level=error msg="Executing migration failed" id="Update is_service_account column to nullable" error="attempt to write a readonly database"
monitoring_grafana | logger=migrator t=2023-02-11T20:48:28.270865226Z level=error msg="Exec failed" error="attempt to write a readonly database" sql="ALTER TABLE user ADD COLUMN tmp_service_account BOOLEAN DEFAULT 0;\nUPDATE user SET tmp_service_account = is_service_account;\nALTER TABLE user DROP COLUMN is_service_account;\nALTER TABLE user RENAME COLUMN tmp_service_account TO is_service_account;"
monitoring_grafana | Failed to start grafana. error: migration failed (id = Update is_service_account column to nullable): attempt to write a readonly database
monitoring_grafana | migration failed (id = Update is_service_account column to nullable): attempt to write a readonly database
prometheus | level=info ts=2023-02-11T20:48:28.386Z caller=main.go:333 msg="Starting Prometheus" version="(version=2.17.1, branch=HEAD, revision=ae041f97cfc6f43494bed65ec4ea4e3a0cf2ac69)"
prometheus | level=info ts=2023-02-11T20:48:28.386Z caller=main.go:334 build_context="(go=go1.13.9, user=root@806b02dfe114, date=20200326-16:18:19)"
prometheus | level=info ts=2023-02-11T20:48:28.386Z caller=main.go:335 host_details="(Linux 4.19.0-23-amd64 #1 SMP Debian 4.19.269-1 (2022-12-20) x86_64 f15eea78067e (none))"
prometheus | level=info ts=2023-02-11T20:48:28.386Z caller=main.go:336 fd_limits="(soft=1048576, hard=1048576)"
prometheus | level=info ts=2023-02-11T20:48:28.386Z caller=main.go:337 vm_limits="(soft=unlimited, hard=unlimited)"
prometheus | level=error ts=2023-02-11T20:48:28.387Z caller=query_logger.go:87 component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied"
prometheus | panic: Unable to create mmap-ed active query log
prometheus |
prometheus | goroutine 1 [running]:
prometheus | github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7ffdc58a4ed5, 0xb, 0x14, 0x2dcaa00, 0xc000523ec0, 0x2dcaa00)
prometheus | /app/promql/query_logger.go:117 +0x4cd
prometheus | main.main()
prometheus | /app/cmd/prometheus/main.go:364 +0x5245
monitoring_grafana exited with code 1
prometheus exited with code 2
monitoring_grafana | logger=settings t=2023-02-11T20:48:29.501095589Z level=info msg="Starting Grafana" version=9.3.6 commit=978237e7cb branch=HEAD compiled=2023-01-26T14:38:58Z
monitoring_grafana | logger=settings t=2023-02-11T20:48:29.501408961Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
monitoring_grafana | logger=settings t=2023-02-11T20:48:29.501434972Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
monitoring_grafana | logger=settings t=2023-02-11T20:48:29.501444029Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
monitoring_grafana | logger=settings t=2023-02-11T20:48:29.501453257Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
monitoring_grafana | logger=settings t=2023-02-11T20:48:29.501466552Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
monitoring_grafana | logger=settings t=2023-02-11T20:48:29.501476792Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
monitoring_grafana | logger=settings t=2023-02-11T20:48:29.501486Z level=info msg="Config overridden from command line" arg="default.log.mode=console"
monitoring_grafana | logger=settings t=2023-02-11T20:48:29.50149648Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana"
monitoring_grafana | logger=settings t=2023-02-11T20:48:29.501505858Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana"
monitoring_grafana | logger=settings t=2023-02-11T20:48:29.501515697Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
monitoring_grafana | logger=settings t=2023-02-11T20:48:29.501520757Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
monitoring_grafana | logger=settings t=2023-02-11T20:48:29.501525796Z level=info msg="Config overridden from Environment variable" var="GF_SECURITY_ADMIN_USER=admin"
monitoring_grafana | logger=settings t=2023-02-11T20:48:29.501531097Z level=info msg="Config overridden from Environment variable" var="GF_SECURITY_ADMIN_PASSWORD=*********"
monitoring_grafana | logger=settings t=2023-02-11T20:48:29.501537048Z level=info msg="Config overridden from Environment variable" var="GF_USERS_ALLOW_SIGN_UP=false"
monitoring_grafana | logger=settings t=2023-02-11T20:48:29.501590761Z level=info msg="Path Home" path=/usr/share/grafana
monitoring_grafana | logger=settings t=2023-02-11T20:48:29.501595431Z level=info msg="Path Data" path=/var/lib/grafana
monitoring_grafana | logger=settings t=2023-02-11T20:48:29.501599248Z level=info msg="Path Logs" path=/var/log/grafana
monitoring_grafana | logger=settings t=2023-02-11T20:48:29.501603055Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
monitoring_grafana | logger=settings t=2023-02-11T20:48:29.501606912Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
monitoring_grafana | logger=settings t=2023-02-11T20:48:29.501611001Z level=info msg="App mode production"
monitoring_grafana | logger=sqlstore t=2023-02-11T20:48:29.501701715Z level=info msg="Connecting to DB" dbtype=sqlite3
monitoring_grafana | logger=sqlstore t=2023-02-11T20:48:29.50174059Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r-----
monitoring_grafana | logger=migrator t=2023-02-11T20:48:29.52129226Z level=info msg="Starting DB migrations"
monitoring_grafana | logger=migrator t=2023-02-11T20:48:29.525709803Z level=info msg="Executing migration" id="Update is_service_account column to nullable"
monitoring_grafana | logger=migrator t=2023-02-11T20:48:29.525819955Z level=error msg="Executing migration failed" id="Update is_service_account column to nullable" error="attempt to write a readonly database"
monitoring_grafana | logger=migrator t=2023-02-11T20:48:29.525834904Z level=error msg="Exec failed" error="attempt to write a readonly database" sql="ALTER TABLE user ADD COLUMN tmp_service_account BOOLEAN DEFAULT 0;\nUPDATE user SET tmp_service_account = is_service_account;\nALTER TABLE user DROP COLUMN is_service_account;\nALTER TABLE user RENAME COLUMN tmp_service_account TO is_service_account;"
monitoring_grafana | Failed to start grafana. error: migration failed (id = Update is_service_account column to nullable): attempt to write a readonly database
monitoring_grafana | migration failed (id = Update is_service_account column to nullable): attempt to write a readonly database
cdn-up-and-running-edge-1 | 2023/02/11 20:48:29 [error] 7#7: *10 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.30.0.1, server: , request: "GET /item_75.ext HTTP/1.1", upstream: "http://172.30.0.2:8080/item_75.ext", host: "localhost:8081"
prometheus | level=info ts=2023-02-11T20:48:29.763Z caller=main.go:333 msg="Starting Prometheus" version="(version=2.17.1, branch=HEAD, revision=ae041f97cfc6f43494bed65ec4ea4e3a0cf2ac69)"
prometheus | level=info ts=2023-02-11T20:48:29.763Z caller=main.go:334 build_context="(go=go1.13.9, user=root@806b02dfe114, date=20200326-16:18:19)"
prometheus | level=info ts=2023-02-11T20:48:29.763Z caller=main.go:335 host_details="(Linux 4.19.0-23-amd64 #1 SMP Debian 4.19.269-1 (2022-12-20) x86_64 f15eea78067e (none))"
prometheus | level=info ts=2023-02-11T20:48:29.763Z caller=main.go:336 fd_limits="(soft=1048576, hard=1048576)"
prometheus | level=info ts=2023-02-11T20:48:29.763Z caller=main.go:337 vm_limits="(soft=unlimited, hard=unlimited)"
prometheus | level=error ts=2023-02-11T20:48:29.764Z caller=query_logger.go:87 component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied"
prometheus | panic: Unable to create mmap-ed active query log
prometheus |
prometheus | goroutine 1 [running]:
prometheus | github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7fff157b3ed5, 0xb, 0x14, 0x2dcaa00, 0xc0007e1ec0, 0x2dcaa00)
prometheus | /app/promql/query_logger.go:117 +0x4cd
prometheus | main.main()
prometheus | /app/cmd/prometheus/main.go:364 +0x5245
cdn-up-and-running-edge-1 | 2023/02/11 20:48:29 [error] 7#7: *1 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.30.0.1, server: , request: "GET /item_24.ext HTTP/1.1", upstream: "http://172.30.0.2:8080/item_24.ext", host: "localhost:8081"
monitoring_grafana exited with code 1
prometheus exited with code 2
monitoring_grafana exited with code 1
prometheus exited with code 2
cdn-up-and-running-edge-1 | 2023/02/11 20:48:30 [error] 7#7: *8 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.30.0.1, server: , request: "GET /item_54.ext HTTP/1.1", upstream: "http://172.30.0.2:8080/item_54.ext", host: "localhost:8081"
cdn-up-and-running-edge-1 | 2023/02/11 20:48:30 [error] 7#7: *6 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.30.0.1, server: , request: "GET /item_98.ext HTTP/1.1", upstream: "http://172.30.0.2:8080/item_98.ext", host: "localhost:8081"
monitoring_grafana | logger=settings t=2023-02-11T20:48:30.988640283Z level=info msg="Starting Grafana" version=9.3.6 commit=978237e7cb branch=HEAD compiled=2023-01-26T14:38:58Z
monitoring_grafana | logger=settings t=2023-02-11T20:48:30.988894152Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
monitoring_grafana | logger=settings t=2023-02-11T20:48:30.988913309Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
monitoring_grafana | logger=settings t=2023-02-11T20:48:30.988919411Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
monitoring_grafana | logger=settings t=2023-02-11T20:48:30.988925091Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
monitoring_grafana | logger=settings t=2023-02-11T20:48:30.988930001Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
monitoring_grafana | logger=settings t=2023-02-11T20:48:30.988935532Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
monitoring_grafana | logger=settings t=2023-02-11T20:48:30.988940592Z level=info msg="Config overridden from command line" arg="default.log.mode=console"
monitoring_grafana | logger=settings t=2023-02-11T20:48:30.988947174Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana"
monitoring_grafana | logger=settings t=2023-02-11T20:48:30.988952996Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana"
monitoring_grafana | logger=settings t=2023-02-11T20:48:30.988957825Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
monitoring_grafana | logger=settings t=2023-02-11T20:48:30.988962694Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
monitoring_grafana | logger=settings t=2023-02-11T20:48:30.988975259Z level=info msg="Config overridden from Environment variable" var="GF_SECURITY_ADMIN_USER=admin"
monitoring_grafana | logger=settings t=2023-02-11T20:48:30.988980909Z level=info msg="Config overridden from Environment variable" var="GF_SECURITY_ADMIN_PASSWORD=*********"
monitoring_grafana | logger=settings t=2023-02-11T20:48:30.988990808Z level=info msg="Config overridden from Environment variable" var="GF_USERS_ALLOW_SIGN_UP=false"
monitoring_grafana | logger=settings t=2023-02-11T20:48:30.98899674Z level=info msg="Path Home" path=/usr/share/grafana
monitoring_grafana | logger=settings t=2023-02-11T20:48:30.989004865Z level=info msg="Path Data" path=/var/lib/grafana
monitoring_grafana | logger=settings t=2023-02-11T20:48:30.989009023Z level=info msg="Path Logs" path=/var/log/grafana
monitoring_grafana | logger=settings t=2023-02-11T20:48:30.989016277Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
monitoring_grafana | logger=settings t=2023-02-11T20:48:30.989020676Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
monitoring_grafana | logger=settings t=2023-02-11T20:48:30.989030163Z level=info msg="App mode production"
monitoring_grafana | logger=sqlstore t=2023-02-11T20:48:30.989114927Z level=info msg="Connecting to DB" dbtype=sqlite3
monitoring_grafana | logger=sqlstore t=2023-02-11T20:48:30.989143101Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r-----
monitoring_grafana | logger=migrator t=2023-02-11T20:48:31.004832178Z level=info msg="Starting DB migrations"
monitoring_grafana | logger=migrator t=2023-02-11T20:48:31.009256503Z level=info msg="Executing migration" id="Update is_service_account column to nullable"
monitoring_grafana | Failed to start grafana. error: migration failed (id = Update is_service_account column to nullable): attempt to write a readonly database
monitoring_grafana | migration failed (id = Update is_service_account column to nullable): attempt to write a readonly database
monitoring_grafana | logger=migrator t=2023-02-11T20:48:31.009359742Z level=error msg="Executing migration failed" id="Update is_service_account column to nullable" error="attempt to write a readonly database"
monitoring_grafana | logger=migrator t=2023-02-11T20:48:31.00936888Z level=error msg="Exec failed" error="attempt to write a readonly database" sql="ALTER TABLE user ADD COLUMN tmp_service_account BOOLEAN DEFAULT 0;\nUPDATE user SET tmp_service_account = is_service_account;\nALTER TABLE user DROP COLUMN is_service_account;\nALTER TABLE user RENAME COLUMN tmp_service_account TO is_service_account;"
monitoring_grafana exited with code 1
cdn-up-and-running-edge-1 | 2023/02/11 20:48:31 [error] 7#7: *4 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.30.0.1, server: , request: "GET /item_60.ext HTTP/1.1", upstream: "http://172.30.0.2:8080/item_60.ext", host: "localhost:8081"
prometheus | level=info ts=2023-02-11T20:48:31.934Z caller=main.go:333 msg="Starting Prometheus" version="(version=2.17.1, branch=HEAD, revision=ae041f97cfc6f43494bed65ec4ea4e3a0cf2ac69)"
prometheus | level=info ts=2023-02-11T20:48:31.934Z caller=main.go:334 build_context="(go=go1.13.9, user=root@806b02dfe114, date=20200326-16:18:19)"
prometheus | level=info ts=2023-02-11T20:48:31.934Z caller=main.go:335 host_details="(Linux 4.19.0-23-amd64 #1 SMP Debian 4.19.269-1 (2022-12-20) x86_64 f15eea78067e (none))"
prometheus | level=info ts=2023-02-11T20:48:31.934Z caller=main.go:336 fd_limits="(soft=1048576, hard=1048576)"
prometheus | level=info ts=2023-02-11T20:48:31.934Z caller=main.go:337 vm_limits="(soft=unlimited, hard=unlimited)"
prometheus | level=error ts=2023-02-11T20:48:31.935Z caller=query_logger.go:87 component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied"
prometheus | panic: Unable to create mmap-ed active query log
prometheus |
prometheus | goroutine 1 [running]:
prometheus | github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7ffd3c753ed5, 0xb, 0x14, 0x2dcaa00, 0xc0007f6cc0, 0x2dcaa00)
prometheus | /app/promql/query_logger.go:117 +0x4cd
prometheus | main.main()
prometheus | /app/cmd/prometheus/main.go:364 +0x5245
I tried reading through prometheus/prometheus#5976, tried a few of the proposed solutions, but nothing worked, so I ended up giving root user permission to prometheus and 1000:1000 user to grafana containers
prometheus:
image: prom/prometheus:v2.17.1
user: root
...
grafana:
image: grafana/grafana:latest
user: "1000:1000"
...
This seems to have fixed the problem. wrk returns
Running 10m test @ http://localhost:8081
2 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 145.64ms 213.07ms 1.98s 84.62%
Req/Sec 49.79 33.66 230.00 70.99%
Latency Distribution
50% 35.95ms
75% 240.63ms
90% 453.02ms
99% 1.01s
29946 requests in 10.00m, 12.32MB read
Socket errors: connect 0, read 13956, write 19020644, timeout 9
Non-2xx or 3xx responses: 334
Requests/sec: 49.91
Transfer/sec: 21.03KB
however the edge logs still print
cdn-up-and-running-edge-1 | 2023/02/11 20:54:06 [error] 7#7: *96 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.30.0.1, server: , request: "GET /item_36.ext HTTP/1.1", upstream: "http://172.30.0.2:8080/item_36.ext", host: "localhost:8081"
on every request which looks fishy but I'm not sure what it means.