[Bug] Unable to restore backup to new Galera cluster
Closed this issue · 23 comments
perfectra1n commented
Documentation
- I acknowledge that I have read the relevant documentation.
Describe the bug
I have a .sql file that was created as a Backup, and I wish to restore it to a newly created Galera cluster.
Expected behaviour
It seems like once the cluster is created, everything works fine. The restore job takes place, and then the MariaDB container in the pods start crashing. I've validated that I'm able to login as root into the MariaDB before the crashing begins, to make sure that the livenessProbe at least works.
Once the pods start crashing, Galera is not able to recover and everything goes into a crash loop.
Steps to reproduce the bug
- Have S3 backup
- Create
spec.bootstrapFrom.s3section - Watch chaos ensue after recovery
Debug information
MariaDB Resource kubectl get events
[root on DESKTOP-M0FBO5A] ~
» kubectl get events --field-selector involvedObject.name=mariadb-gnew -n mariadb
LAST SEEN TYPE REASON OBJECT MESSAGE
7m17s Normal NoPods poddisruptionbudget/mariadb-gnew No matching pods found
7m17s Normal SuccessfulCreate statefulset/mariadb-gnew create Claim galera-mariadb-gnew-0 Pod mariadb-gnew-0 in StatefulSet mariadb-gnew success
7m17s Normal SuccessfulCreate statefulset/mariadb-gnew create Pod mariadb-gnew-0 in StatefulSet mariadb-gnew successful
7m17s Normal SuccessfulCreate statefulset/mariadb-gnew create Claim storage-mariadb-gnew-2 Pod mariadb-gnew-2 in StatefulSet mariadb-gnew success
7m17s Normal SuccessfulCreate statefulset/mariadb-gnew create Claim storage-mariadb-gnew-1 Pod mariadb-gnew-1 in StatefulSet mariadb-gnew success
7m17s Normal SuccessfulCreate statefulset/mariadb-gnew create Claim galera-mariadb-gnew-2 Pod mariadb-gnew-2 in StatefulSet mariadb-gnew success
7m17s Normal SuccessfulCreate statefulset/mariadb-gnew create Claim galera-mariadb-gnew-1 Pod mariadb-gnew-1 in StatefulSet mariadb-gnew success
7m17s Normal SuccessfulCreate statefulset/mariadb-gnew create Pod mariadb-gnew-1 in StatefulSet mariadb-gnew successful
7m17s Normal SuccessfulCreate statefulset/mariadb-gnew create Pod mariadb-gnew-2 in StatefulSet mariadb-gnew successful
7m1s Normal IPAllocated service/mariadb-gnew Assigned IP ["10.11.0.30"]
4m11s Normal nodeAssigned service/mariadb-gnew announcing from node "newworkernode2" with protocol "bgp"
4m11s Normal nodeAssigned service/mariadb-gnew announcing from node "newworkernode1" with protocol "bgp"
4m11s Normal nodeAssigned service/mariadb-gnew announcing from node "newworkernode3" with protocol "bgp"
4m11s Normal nodeAssigned service/mariadb-gnew announcing from node "newworkernode4" with protocol "bgp"
4m11s Normal nodeAssigned service/mariadb-gnew announcing from node "newworkernode5" with protocol "bgp"
5m51s Normal GaleraClusterHealthy mariadb/mariadb-gnew Galera cluster is healthy
4m33s Normal PrimarySwitching mariadb/mariadb-gnew Switching primary from index '0' to index '1'
4m33s Normal PrimarySwitched mariadb/mariadb-gnew Primary switched from index '0' to index '1'
4m11s Normal PrimarySwitching mariadb/mariadb-gnew Switching primary from index '1' to index '2'
4m11s Normal PrimarySwitched mariadb/mariadb-gnew Primary switched from index '1' to index '2'Full Log of a Pod with crash
{"level":"info","ts":1713901112.128137,"msg":"Starting agent"}
{"level":"info","ts":1713901112.13175,"logger":"server","msg":"server listening","addr":":5555"}
2024/04/23 19:39:13 "DELETE http://mariadb-gnew-1.mariadb-gnew-internal.mariadb.svc.newcluster.local:5555/api/bootstrap HTTP/1.1" from 10.233.69.21:44386 - 404 41B in 11.645103ms
{"level":"error","ts":1713901162.1703997,"logger":"handler.probe.readiness","msg":"error getting MariaDB","error":"Get \"https://10.233.0.1:443/apis/k8s.mariadb.com/v1alpha1/namespaces/mariadb/mariadbs/mariadb-gnew\": context deadline exceeded","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:96\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901221.1846123,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901221.1885583,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901226.184601,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901226.2316885,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901231.182878,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901231.186301,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901256.190096,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901256.1904545,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901261.1816254,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901261.1843452,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901266.1811845,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901266.1869998,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901266.2037497,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901291.184306,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901291.1846526,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901296.1825287,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901296.1837292,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901301.1870213,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901301.187811,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
[mysql] 2024/04/23 19:41:42 packets.go:37: read tcp [::1]:35878->[::1]:3306: read: connection reset by peer
{"level":"error","ts":1713901302.220712,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"dial tcp [::1]:3306: connect: connection refused","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901303.6034021,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"dial tcp [::1]:3306: connect: connection refused","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901326.1832745,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901326.185197,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901331.1809835,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901331.1821089,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901336.1823082,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901336.1838481,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901336.199937,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901339.0055046,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"dial tcp [::1]:3306: connect: connection refused","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
2024-04-23 19:48:02+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.3.2+maria~ubu2204 started.
2024-04-23 19:48:02+00:00 [Warn] [Entrypoint]: /sys/fs/cgroup///memory.pressure not writable, functionality unavailable to MariaDB
2024-04-23 19:48:02+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2024-04-23 19:48:02+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.3.2+maria~ubu2204 started.
2024-04-23 19:48:03+00:00 [Note] [Entrypoint]: MariaDB upgrade information missing, assuming required
2024-04-23 19:48:03+00:00 [Note] [Entrypoint]: MariaDB upgrade (mariadb-upgrade or creating healthcheck users) required, but skipped due to $MARIADB_AUTO_UPGRADE setting
2024-04-23 19:48:03 0 [Note] Starting MariaDB 11.3.2-MariaDB-1:11.3.2+maria~ubu2204 source revision 068a6819eb63bcb01fdfa037c9bf3bf63c33ee42 as process 1
2024-04-23 19:48:03 0 [Note] WSREP: Loading provider /usr/lib/galera/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1
2024-04-23 19:48:03 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
2024-04-23 19:48:03 0 [Note] WSREP: wsrep_load(): Galera 26.4.16(r7dce5149) by Codership Oy <info@codership.com> loaded successfully.
2024-04-23 19:48:03 0 [Note] WSREP: Initializing allowlist service v1
2024-04-23 19:48:03 0 [Note] WSREP: Initializing event service v1
2024-04-23 19:48:03 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.
2024-04-23 19:48:03 0 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 0
2024-04-23 19:48:03 0 [Note] WSREP: GCache DEBUG: opened preamble:
Version: 2
UUID: 003b272f-01a9-11ef-beba-026f3aace78f
Seqno: -1 - -1
Offset: -1
Synced: 0
2024-04-23 19:48:03 0 [Note] WSREP: Recovering GCache ring buffer: version: 2, UUID: 003b272f-01a9-11ef-beba-026f3aace78f, offset: -1
2024-04-23 19:48:03 0 [Note] WSREP: GCache::RingBuffer initial scan... 0.0% ( 0/134217752 bytes) complete.
2024-04-23 19:48:03 0 [Note] WSREP: GCache::RingBuffer initial scan...100.0% (134217752/134217752 bytes) complete.
2024-04-23 19:48:03 0 [Note] WSREP: Recovering GCache ring buffer: found gapless sequence 675-1409
2024-04-23 19:48:03 0 [Note] WSREP: GCache::RingBuffer unused buffers scan... 0.0% ( 0/132302552 bytes) complete.
2024-04-23 19:48:03 0 [Note] WSREP: Recovering GCache ring buffer: found 7/742 locked buffers
2024-04-23 19:48:03 0 [Note] WSREP: Recovering GCache ring buffer: free space: 1916680/134217728
2024-04-23 19:48:03 0 [Note] WSREP: GCache::RingBuffer unused buffers scan...100.0% (132302552/132302552 bytes) complete.
2024-04-23 19:48:03 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.233.91.138; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.keep_plaintext_size = 128M; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0
2024-04-23 19:48:03 0 [Note] WSREP: Start replication
2024-04-23 19:48:03 0 [Note] WSREP: Connecting with bootstrap option: 0
2024-04-23 19:48:03 0 [Note] WSREP: Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1
2024-04-23 19:48:03 0 [Note] WSREP: protonet asio version 0
2024-04-23 19:48:03 0 [Note] WSREP: Using CRC-32C for message checksums.
2024-04-23 19:48:03 0 [Note] WSREP: backend: asio
2024-04-23 19:48:03 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
2024-04-23 19:48:03 0 [Note] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
2024-04-23 19:48:03 0 [Note] WSREP: restore pc from disk failed
2024-04-23 19:48:03 0 [Note] WSREP: GMCast version 0
2024-04-23 19:48:03 0 [Note] WSREP: (65c7b769-a4b9, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
2024-04-23 19:48:03 0 [Note] WSREP: (65c7b769-a4b9, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
2024-04-23 19:48:03 0 [Note] WSREP: EVS version 1
2024-04-23 19:48:03 0 [Note] WSREP: gcomm: connecting to group 'mariadb-operator', peer 'mariadb-gnew-0.mariadb-gnew-internal.mariadb.svc.newcluster.local:,mariadb-gnew-1.mariadb-gnew-internal.mariadb.svc.newcluster.local:,mariadb-gnew-2.mariadb-gnew-internal.mariadb.svc.newcluster.local:'
2024-04-23 19:48:03 0 [Note] WSREP: (65c7b769-a4b9, 'tcp://0.0.0.0:4567') Found matching local endpoint for a connection, blacklisting address tcp://10.233.91.138:4567
2024-04-23 19:48:03 0 [Note] WSREP: (65c7b769-a4b9, 'tcp://0.0.0.0:4567') connection established to 1f792853-896d tcp://10.233.101.184:4567
2024-04-23 19:48:03 0 [Note] WSREP: (65c7b769-a4b9, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2024-04-23 19:48:04 0 [Note] WSREP: EVS version upgrade 0 -> 1
2024-04-23 19:48:04 0 [Note] WSREP: declaring 1f792853-896d at tcp://10.233.101.184:4567 stable
2024-04-23 19:48:04 0 [Note] WSREP: PC protocol upgrade 0 -> 1
2024-04-23 19:48:04 0 [Note] WSREP: Node 1f792853-896d state prim
2024-04-23 19:48:04 0 [Note] WSREP: view(view_id(PRIM,1f792853-896d,22) memb {
1f792853-896d,0
65c7b769-a4b9,0
} joined {
} left {
} partitioned {
})
2024-04-23 19:48:04 0 [Note] WSREP: save pc into disk
2024-04-23 19:48:04 0 [Note] WSREP: discarding pending addr without UUID: tcp://10.233.69.10:4567
2024-04-23 19:48:04 0 [Note] WSREP: gcomm: connected
2024-04-23 19:48:04 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
2024-04-23 19:48:04 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
2024-04-23 19:48:04 0 [Note] WSREP: Opened channel 'mariadb-operator'
2024-04-23 19:48:04 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
2024-04-23 19:48:04 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
2024-04-23 19:48:04 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 66165298-01aa-11ef-91b2-43347ae01c3b
2024-04-23 19:48:04 0 [Note] WSREP: STATE EXCHANGE: got state msg: 66165298-01aa-11ef-91b2-43347ae01c3b from 0 (mariadb-gnew-2)
2024-04-23 19:48:04 0 [Note] WSREP: Initializing config service v1
2024-04-23 19:48:04 1 [Note] WSREP: Starting rollbacker thread 1
2024-04-23 19:48:04 2 [Note] WSREP: Starting applier thread 2
2024-04-23 19:48:04 0 [Note] WSREP: STATE EXCHANGE: got state msg: 66165298-01aa-11ef-91b2-43347ae01c3b from 1 (mariadb-gnew-1)
2024-04-23 19:48:04 0 [Note] WSREP: Quorum results:
version = 6,
component = PRIMARY,
conf_id = 20,
members = 1/2 (joined/total),
act_id = 1416,
last_appl. = 1284,
protocols = 2/10/4 (gcs/repl/appl),
vote policy= 0,
group UUID = 003b272f-01a9-11ef-beba-026f3aace78f
2024-04-23 19:48:04 0 [Note] WSREP: Flow-control interval: [23, 23]
2024-04-23 19:48:04 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 1417)
2024-04-23 19:48:04 0 [Note] WSREP: Deinitializing config service v1
2024-04-23 19:48:04 2 [Note] WSREP: ####### processing CC 1417, local, ordered
2024-04-23 19:48:04 2 [Note] WSREP: Process first view: 003b272f-01a9-11ef-beba-026f3aace78f my uuid: 65c7b769-01aa-11ef-a4b9-4fcbab8908cb
2024-04-23 19:48:04 2 [Note] WSREP: Server mariadb-gnew-1 connected to cluster at position 003b272f-01a9-11ef-beba-026f3aace78f:1417 with ID 65c7b769-01aa-11ef-a4b9-4fcbab8908cb
2024-04-23 19:48:04 2 [Note] WSREP: Server status change disconnected -> connected
2024-04-23 19:48:04 2 [Note] WSREP: ####### My UUID: 65c7b769-01aa-11ef-a4b9-4fcbab8908cb
2024-04-23 19:48:04 2 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes
2024-04-23 19:48:04 0 [Note] WSREP: Service thread queue flushed.
2024-04-23 19:48:04 2 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
2024-04-23 19:48:04 2 [Note] WSREP: State transfer required:
Group state: 003b272f-01a9-11ef-beba-026f3aace78f:1417
Local state: 00000000-0000-0000-0000-000000000000:-1
2024-04-23 19:48:04 2 [Note] WSREP: Server status change connected -> joiner
2024-04-23 19:48:04 0 [Note] WSREP: Joiner monitor thread started to monitor
2024-04-23 19:48:04 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.233.91.138' --datadir '/var/lib/mysql/' --parent 1 --progress 0'
WSREP_SST: [INFO] mariabackup SST started on joiner (20240423 19:48:04.639)
WSREP_SST: [INFO] SSL configuration: CA='', CAPATH='', CERT='', KEY='', MODE='DISABLED', encrypt='0' (20240423 19:48:04.791)
WSREP_SST: [INFO] Progress reporting tool pv not found in path: /usr//bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/sbin:/usr/bin:/sbin:/bin (20240423 19:48:05.150)
WSREP_SST: [INFO] Disabling all progress/rate-limiting (20240423 19:48:05.154)
WSREP_SST: [INFO] Streaming with mbstream (20240423 19:48:05.187)
WSREP_SST: [INFO] Using socat as streamer (20240423 19:48:05.191)
WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql/sst_in_progress (20240423 19:48:05.197)
WSREP_SST: [INFO] previous SST is not completed, waiting for it to exit (20240423 19:48:05.252)
WSREP_SST: [INFO] previous SST is not completed, waiting for it to exit (20240423 19:48:06.283)
2024-04-23 19:48:07 0 [Note] WSREP: (65c7b769-a4b9, 'tcp://0.0.0.0:4567') turning message relay requesting off
WSREP_SST: [INFO] previous SST is not completed, waiting for it to exit (20240423 19:48:07.315)
WSREP_SST: [INFO] previous SST is not completed, waiting for it to exit (20240423 19:48:08.350)
WSREP_SST: [INFO] previous SST is not completed, waiting for it to exit (20240423 19:48:09.381)
WSREP_SST: [INFO] previous SST is not completed, waiting for it to exit (20240423 19:48:10.423)
WSREP_SST: [INFO] previous SST is not completed, waiting for it to exit (20240423 19:48:11.458)
WSREP_SST: [INFO] previous SST is not completed, waiting for it to exit (20240423 19:48:12.500)
WSREP_SST: [INFO] previous SST is not completed, waiting for it to exit (20240423 19:48:13.536)
WSREP_SST: [INFO] previous SST is not completed, waiting for it to exit (20240423 19:48:14.574)
WSREP_SST: [ERROR] previous SST script still running. (20240423 19:48:14.584)
2024-04-23 19:48:14 0 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_mariabackup --role 'joiner' --address '10.233.91.138' --datadir '/var/lib/mysql/' --parent 1 --progress 0
Read: '(null)'
2024-04-23 19:48:14 0 [ERROR] WSREP: Process completed with error: wsrep_sst_mariabackup --role 'joiner' --address '10.233.91.138' --datadir '/var/lib/mysql/' --parent 1 --progress 0: 114 (Operation already in progress)
2024-04-23 19:48:14 2 [ERROR] WSREP: Failed to prepare for 'mariabackup' SST. Unrecoverable.
2024-04-23 19:48:14 2 [ERROR] WSREP: SST request callback failed. This is unrecoverable, restart required.
2024-04-23 19:48:14 2 [Note] WSREP: ReplicatorSMM::abort()
2024-04-23 19:48:14 2 [Note] WSREP: Closing send monitor...
2024-04-23 19:48:14 2 [Note] WSREP: Closed send monitor.
2024-04-23 19:48:14 2 [Note] WSREP: gcomm: terminating thread
2024-04-23 19:48:14 2 [Note] WSREP: gcomm: joining thread
2024-04-23 19:48:14 2 [Note] WSREP: gcomm: closing backend
2024-04-23 19:48:15 2 [Note] WSREP: view(view_id(NON_PRIM,1f792853-896d,22) memb {
65c7b769-a4b9,0
} joined {
} left {
} partitioned {
1f792853-896d,0
})
2024-04-23 19:48:15 2 [Note] WSREP: PC protocol downgrade 1 -> 0
2024-04-23 19:48:15 2 [Note] WSREP: view((empty))
2024-04-23 19:48:15 2 [Note] WSREP: gcomm: closed
2024-04-23 19:48:15 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2024-04-23 19:48:15 0 [Note] WSREP: Flow-control interval: [16, 16]
2024-04-23 19:48:15 0 [Note] WSREP: Received NON-PRIMARY.
2024-04-23 19:48:15 0 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 1417)
2024-04-23 19:48:15 0 [Note] WSREP: New SELF-LEAVE.
2024-04-23 19:48:15 0 [Note] WSREP: Flow-control interval: [0, 0]
2024-04-23 19:48:15 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2024-04-23 19:48:15 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 1417)
2024-04-23 19:48:15 0 [Note] WSREP: RECV thread exiting 0: Success
2024-04-23 19:48:15 2 [Note] WSREP: recv_thread() joined.
2024-04-23 19:48:15 2 [Note] WSREP: Closing replication queue.
2024-04-23 19:48:15 2 [Note] WSREP: Closing slave action queue.
2024-04-23 19:48:15 2 [Note] WSREP: mariadbd: Terminated.
240423 19:48:15 [ERROR] mysqld got signal 11 ;
Sorry, we probably made a mistake, and this is a bug.
Your assistance in bug reporting will enable us to fix this for the next release.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 11.3.2-MariaDB-1:11.3.2+maria~ubu2204 source revision: 068a6819eb63bcb01fdfa037c9bf3bf63c33ee42
key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=3
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 336992 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7f6f5c000c68
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f6f81425c68 thread_stack 0x49000
Printing to addr2line failed
mariadbd(my_print_stacktrace+0x32)[0x55a1a86358a2]
mariadbd(handle_fatal_signal+0x478)[0x55a1a8106488]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f6f83bef520]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x178)[0x7f6f83bd5898]
/usr/lib/galera/libgalera_smm.so(+0x157602)[0x7f6f83671602]
/usr/lib/galera/libgalera_smm.so(+0x700e1)[0x7f6f8358a0e1]
/usr/lib/galera/libgalera_smm.so(+0x6cc94)[0x7f6f83586c94]
/usr/lib/galera/libgalera_smm.so(+0x8b311)[0x7f6f835a5311]
/usr/lib/galera/libgalera_smm.so(+0x604a0)[0x7f6f8357a4a0]
/usr/lib/galera/libgalera_smm.so(+0x48261)[0x7f6f83562261]
mariadbd(_ZN5wsrep18wsrep_provider_v2611run_applierEPNS_21high_priority_serviceE+0x12)[0x55a1a86f5592]
mariadbd(+0xd93e31)[0x55a1a83c5e31]
mariadbd(_Z15start_wsrep_THDPv+0x26b)[0x55a1a83b3a7b]
mariadbd(+0xd05f86)[0x55a1a8337f86]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f6f83c41ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f6f83cd3850]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0): (null)
Connection ID (thread ID): 2
Status: NOT_KILLED
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off,hash_join_cardinality=on,cset_narrowing=off,sargable_casefold=on
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/ contains
information that should help you find out what is causing the crash.
We think the query pointer is invalid, but we will try to print it anyway.
Query:
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 0 bytes
Max resident set unlimited unlimited bytes
Max processes unlimited unlimited processes
Max open files 65535 65535 files
Max locked memory unlimited unlimited bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 192953 192953 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Core pattern: core
Kernel version: Linux version 5.10.0-28-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.209-2 (2024-01-31)
perfectra1n commented
mariadb-operator Debug Logs
{"level":"debug","ts":1713901054.6608806,"msg":"error getting StatefulSet","controller":"mariadb","controllerGroup":"k8s.mariadb.com","controllerKind":"MariaDB","MariaDB":{"name":"mariadb-gnew","namespace":"mariadb"},"namespace":"mariadb","name":"mariadb-gnew","reconcileID":"96724d12-60f1-4fc7-989b-9f25ce450930","err":"StatefulSet.apps \"mariadb-gnew\" not found"}
{"level":"debug","ts":1713901054.660993,"msg":"error getting init Job","controller":"mariadb","controllerGroup":"k8s.mariadb.com","controllerKind":"MariaDB","MariaDB":{"name":"mariadb-gnew","namespace":"mariadb"},"namespace":"mariadb","name":"mariadb-gnew","reconcileID":"96724d12-60f1-4fc7-989b-9f25ce450930","err":"Job.batch \"mariadb-gnew-init\" not found"}
{"level":"debug","ts":1713901054.8712292,"msg":"Creating init job","controller":"mariadb","controllerGroup":"k8s.mariadb.com","controllerKind":"MariaDB","MariaDB":{"name":"mariadb-gnew","namespace":"mariadb"},"namespace":"mariadb","name":"mariadb-gnew","reconcileID":"96724d12-60f1-4fc7-989b-9f25ce450930"}
{"level":"debug","ts":1713901054.9329562,"msg":"error getting StatefulSet","controller":"mariadb","controllerGroup":"k8s.mariadb.com","controllerKind":"MariaDB","MariaDB":{"name":"mariadb-gnew","namespace":"mariadb"},"namespace":"mariadb","name":"mariadb-gnew","reconcileID":"20aa67cb-c49f-4b5d-bb0b-2afcbab11eb9","err":"StatefulSet.apps \"mariadb-gnew\" not found"}
{"level":"debug","ts":1713901054.9586332,"msg":"Init job not completed. Requeuing","controller":"mariadb","controllerGroup":"k8s.mariadb.com","controllerKind":"MariaDB","MariaDB":{"name":"mariadb-gnew","namespace":"mariadb"},"namespace":"mariadb","name":"mariadb-gnew","reconcileID":"20aa67cb-c49f-4b5d-bb0b-2afcbab11eb9"}
{"level":"debug","ts":1713901055.0108721,"msg":"error getting StatefulSet","controller":"mariadb","controllerGroup":"k8s.mariadb.com","controllerKind":"MariaDB","MariaDB":{"name":"mariadb-gnew","namespace":"mariadb"},"namespace":"mariadb","name":"mariadb-gnew","reconcileID":"48ca5460-272c-40e8-b5a4-bc84dc09e0ff","err":"StatefulSet.apps \"mariadb-gnew\" not found"}
{"level":"debug","ts":1713901055.038942,"msg":"Init job not completed. Requeuing","controller":"mariadb","controllerGroup":"k8s.mariadb.com","controllerKind":"MariaDB","MariaDB":{"name":"mariadb-gnew","namespace":"mariadb"},"namespace":"mariadb","name":"mariadb-gnew","reconcileID":"48ca5460-272c-40e8-b5a4-bc84dc09e0ff"}
{"level":"debug","ts":1713901055.9270701,"msg":"error getting StatefulSet","controller":"mariadb","controllerGroup":"k8s.mariadb.com","controllerKind":"MariaDB","MariaDB":{"name":"mariadb-gnew","namespace":"mariadb"},"namespace":"mariadb","name":"mariadb-gnew","reconcileID":"187296f9-184c-4def-b545-127e9af25f81","err":"StatefulSet.apps \"mariadb-gnew\" not found"}
{"level":"debug","ts":1713901055.95898,"msg":"Init job not completed. Requeuing","controller":"mariadb","controllerGroup":"k8s.mariadb.com","controllerKind":"MariaDB","MariaDB":{"name":"mariadb-gnew","namespace":"mariadb"},"namespace":"mariadb","name":"mariadb-gnew","reconcileID":"187296f9-184c-4def-b545-127e9af25f81"}
{"level":"debug","ts":1713901057.0024385,"msg":"error getting StatefulSet","controller":"mariadb","controllerGroup":"k8s.mariadb.com","controllerKind":"MariaDB","MariaDB":{"name":"mariadb-gnew","namespace":"mariadb"},"namespace":"mariadb","name":"mariadb-gnew","reconcileID":"63f877bd-3f82-4f22-ac98-17d836f87093","err":"StatefulSet.apps \"mariadb-gnew\" not found"}
{"level":"debug","ts":1713901057.0302658,"msg":"Init job not completed. Requeuing","controller":"mariadb","controllerGroup":"k8s.mariadb.com","controllerKind":"MariaDB","MariaDB":{"name":"mariadb-gnew","namespace":"mariadb"},"namespace":"mariadb","name":"mariadb-gnew","reconcileID":"63f877bd-3f82-4f22-ac98-17d836f87093"}
{"level":"debug","ts":1713901058.0933204,"msg":"error getting StatefulSet","controller":"mariadb","controllerGroup":"k8s.mariadb.com","controllerKind":"MariaDB","MariaDB":{"name":"mariadb-gnew","namespace":"mariadb"},"namespace":"mariadb","name":"mariadb-gnew","reconcileID":"4d76b98a-93e3-496d-a15c-cb820a9788c1","err":"StatefulSet.apps \"mariadb-gnew\" not found"}
{"level":"debug","ts":1713901058.120356,"msg":"Init job not completed. Requeuing","controller":"mariadb","controllerGroup":"k8s.mariadb.com","controllerKind":"MariaDB","MariaDB":{"name":"mariadb-gnew","namespace":"mariadb"},"namespace":"mariadb","name":"mariadb-gnew","reconcileID":"4d76b98a-93e3-496d-a15c-cb820a9788c1"}
{"level":"debug","ts":1713901059.164477,"msg":"error getting StatefulSet","controller":"mariadb","controllerGroup":"k8s.mariadb.com","controllerKind":"MariaDB","MariaDB":{"name":"mariadb-gnew","namespace":"mariadb"},"namespace":"mariadb","name":"mariadb-gnew","reconcileID":"ad7a953e-0bd2-4f5f-8d6e-47e72eb084d9","err":"StatefulSet.apps \"mariadb-gnew\" not found"}
{"level":"debug","ts":1713901059.1945257,"msg":"Init job not completed. Requeuing","controller":"mariadb","controllerGroup":"k8s.mariadb.com","controllerKind":"MariaDB","MariaDB":{"name":"mariadb-gnew","namespace":"mariadb"},"namespace":"mariadb","name":"mariadb-gnew","reconcileID":"ad7a953e-0bd2-4f5f-8d6e-47e72eb084d9"}
{"level":"debug","ts":1713901060.264981,"msg":"error getting StatefulSet","controller":"mariadb","controllerGroup":"k8s.mariadb.com","controllerKind":"MariaDB","MariaDB":{"name":"mariadb-gnew","namespace":"mariadb"},"namespace":"mariadb","name":"mariadb-gnew","reconcileID":"72b32d47-c175-4754-a7a2-1e9b4b0f1ab0","err":"StatefulSet.apps \"mariadb-gnew\" not found"}
{"level":"debug","ts":1713901060.2998955,"msg":"Init job not completed. Requeuing","controller":"mariadb","controllerGroup":"k8s.mariadb.com","controllerKind":"MariaDB","MariaDB":{"name":"mariadb-gnew","namespace":"mariadb"},"namespace":"mariadb","name":"mariadb-gnew","reconcileID":"72b32d47-c175-4754-a7a2-1e9b4b0f1ab0"}
{"level":"debug","ts":1713901061.344289,"msg":"error getting StatefulSet","controller":"mariadb","controllerGroup":"k8s.mariadb.com","controllerKind":"MariaDB","MariaDB":{"name":"mariadb-gnew","namespace":"mariadb"},"namespace":"mariadb","name":"mariadb-gnew","reconcileID":"830fba35-44c7-4e45-b5db-99e7a551e4e0","err":"StatefulSet.apps \"mariadb-gnew\" not found"}
{"level":"debug","ts":1713901061.3713527,"msg":"Init job not completed. Requeuing","controller":"mariadb","controllerGroup":"k8s.mariadb.com","controllerKind":"MariaDB","MariaDB":{"name":"mariadb-gnew","namespace":"mariadb"},"namespace":"mariadb","name":"mariadb-gnew","reconcileID":"830fba35-44c7-4e45-b5db-99e7a551e4e0"}
{"level":"debug","ts":1713901062.4126098,"msg":"error getting StatefulSet","controller":"mariadb","controllerGroup":"k8s.mariadb.com","controllerKind":"MariaDB","MariaDB":{"name":"mariadb-gnew","namespace":"mariadb"},"namespace":"mariadb","name":"mariadb-gnew","reconcileID":"60397af5-2672-485f-ab00-344017a5b65b","err":"StatefulSet.apps \"mariadb-gnew\" not found"}
{"level":"debug","ts":1713901062.444595,"msg":"Init job not completed. Requeuing","controller":"mariadb","controllerGroup":"k8s.mariadb.com","controllerKind":"MariaDB","MariaDB":{"name":"mariadb-gnew","namespace":"mariadb"},"namespace":"mariadb","name":"mariadb-gnew","reconcileID":"60397af5-2672-485f-ab00-344017a5b65b"}
{"level":"debug","ts":1713901063.488736,"msg":"error getting StatefulSet","controller":"mariadb","controllerGroup":"k8s.mariadb.com","controllerKind":"MariaDB","MariaDB":{"name":"mariadb-gnew","namespace":"mariadb"},"namespace":"mariadb","name":"mariadb-gnew","reconcileID":"1b9b4ee7-2f55-4ed4-aa36-3686e23476eb","err":"StatefulSet.apps \"mariadb-gnew\" not found"}Environment details:
- Kubernetes version: v1.28.6
- Kubernetes distribution: Vanilla
- mariadb-operator version: 0.27.0
- Install method: Helm
- Install flavor: Recommended
Additional context
MariaDB YAML
apiVersion: k8s.mariadb.com/v1alpha1
kind: MariaDB
metadata:
name: mariadb-gnew
namespace: mariadb
annotations:
argocd.argoproj.io/compare-options: IgnoreExtraneous
argocd.argoproj.io/sync-options: Prune=false
spec:
rootPasswordSecretKeyRef:
name: mariadb-creds
key: root-password
podSecurityContext:
runAsUser: 0
storage:
size: 30Gi
storageClassName: local-path
resizeInUseVolumes: true
waitForVolumeResize: true
volumeClaimTemplate:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 30Gi
storageClassName: local-path
image: mariadb:11.3.2
replicas: 3
bootstrapFrom:
s3:
bucket: mysql-backups
endpoint: minio.minio.svc.newcluster.local:9000
accessKeyIdSecretKeyRef:
name: minio-creds
key: MINIO_ACCESS_KEY
secretAccessKeySecretKeyRef:
name: minio-creds
key: MINIO_SECRET_KEY
tls:
enabled: false
targetRecoveryTime: 2024-04-22T02:00:00Z
galera:
enabled: true
primary:
automaticFailover: true
replicaThreads: 1
agent:
image: ghcr.io/mariadb-operator/mariadb-operator:v0.0.27
port: 5555
kubernetesAuth:
enabled: true
gracefulShutdownTimeout: 1s
recovery:
enabled: true
minClusterSize: 40%
clusterHealthyTimeout: 30s
clusterBootstrapTimeout: 10m0s
podRecoveryTimeout: 3m0s
podSyncTimeout: 3m0s
initContainer:
image: ghcr.io/mariadb-operator/mariadb-operator:v0.0.27
initJob:
labels:
sidecar.istio.io/inject: "false"
config:
reuseStorageVolume: false
volumeClaimTemplate:
resources:
requests:
storage: 300Mi
accessModes:
- ReadWriteOnce
service:
type: LoadBalancer
annotations:
metallb.universe.tf/ip-allocated-from-pool: first-pool
metallb.universe.tf/loadBalancerIPs: 10.11.0.30
connection:
secretName: mariadb-gnew-conn
secretTemplate:
key: dsn
primaryService:
type: LoadBalancer
annotations:
metallb.universe.tf/ip-allocated-from-pool: first-pool
metallb.universe.tf/loadBalancerIPs: 10.11.0.29
primaryConnection:
secretName: mariadb-gnew-conn-primary
secretTemplate:
key: dsn
secondaryService:
type: LoadBalancer
annotations:
metallb.universe.tf/ip-allocated-from-pool: first-pool
metallb.universe.tf/loadBalancerIPs: 10.11.0.28
secondaryConnection:
secretName: mariadb-gnew-conn-secondary
secretTemplate:
key: dsn
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- topologyKey: "kubernetes.io/hostname"
tolerations:
- key: "mariadb.mmontes.io/ha"
operator: "Exists"
effect: "NoSchedule"
updateStrategy:
type: RollingUpdate
myCnf: |
[mariadb]
bind-address=*
default_storage_engine=InnoDB
binlog_format=row
innodb_autoinc_lock_mode=2
max_allowed_packet=256M
resources:
requests:
cpu: 300m
memory: 256Mi
limits:
memory: 1Gi
metrics:
enabled: truemariadb-operator Deployment YAML
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: mariadb-op
namespace: argocd
spec:
project: default
source:
chart: mariadb-operator
repoURL: https://mariadb-operator.github.io/mariadb-operator
targetRevision: "0.27.0"
helm:
values: |
clusterName: "newcluster.local"
ha:
enabled: true
logLevel: DEBUG
metrics:
enabled: true
webhook:
cert:
certManager:
enabled: true
# This was just erroring over and over again about being null
# https://github.com/mariadb-operator/mariadb-operator/commit/e49eb862ac55e67dd7e7e325ff10180c5ab8fabf
secretLabels:
key1: value1
destination:
server: "https://kubernetes.default.svc"
namespace: mariadb
syncPolicy:
managedNamespaceMetadata:
labels:
goldilocks.fairwinds.com/enabled: "true"
syncOptions:
- CreateNamespace=true
- ServerSideApply=true
automated:
prune: true
selfHeal: true
ignoreDifferences:
- group: apps
kind: Service
jqPathExpressions:
- .metadata.resourceVersion
jsonPointers:
- /metadata/resourceVersionperfectra1n commented
I started this with removing every last CRD and resource associated with mariadb-operator, and then installing everything "fresh" via Helm into the mariadb namespace. This deployment and log is after doing so.
perfectra1n commented
My thought process is the following:
Create MariaDB resource with the following sections
spec.galera
galera:
enabled: true
primary:
automaticFailover: true
replicaThreads: 1
agent:
image: ghcr.io/mariadb-operator/mariadb-operator:v0.0.27
port: 5555
kubernetesAuth:
enabled: true
gracefulShutdownTimeout: 1s
recovery:
enabled: true
minClusterSize: 40%
clusterHealthyTimeout: 30s
clusterBootstrapTimeout: 10m0s
podRecoveryTimeout: 3m0s
podSyncTimeout: 3m0s
initContainer:
image: ghcr.io/mariadb-operator/mariadb-operator:v0.0.27
initJob:
labels:
sidecar.istio.io/inject: "false"
config:
reuseStorageVolume: false
volumeClaimTemplate:
resources:
requests:
storage: 300Mi
accessModes:
- ReadWriteOncespec.boostrapFrom.s3
bootstrapFrom:
s3:
bucket: mysql-backups
endpoint: minio.minio.svc.newcluster.local:9000
accessKeyIdSecretKeyRef:
name: minio-creds
key: MINIO_ACCESS_KEY
secretAccessKeySecretKeyRef:
name: minio-creds
key: MINIO_SECRET_KEY
tls:
enabled: false
targetRecoveryTime: 2024-04-22T02:00:00Zand then the restore should be good, but it doesn't appear to work, and just nukes Galera for some reason 🤷♂️
If I'm doing something wrong or silly, please feel free to let me know. I've spent ~24 hours on this at this point, just trying to get my MariaDB to recover.
perfectra1n commented
I'm also getting logs like the following:
Access denied for user 'root'@'localhost'
{"level":"error","ts":1713901291.1846526,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901296.1825287,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901296.1837292,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901301.1870213,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901301.187811,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
[mysql] 2024/04/23 19:41:42 packets.go:37: read tcp [::1]:35878->[::1]:3306: read: connection reset by peer
{"level":"error","ts":1713901302.220712,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"dial tcp [::1]:3306: connect: connection refused","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901303.6034021,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"dial tcp [::1]:3306: connect: connection refused","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901326.1832745,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901326.185197,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901331.1809835,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901331.1821089,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901336.1823082,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901336.1838481,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901336.199937,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901339.0055046,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"dial tcp [::1]:3306: connect: connection refused","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}However, when copy and pasting the secret it seems to be just fine:
root@mariadb-gnew-2:/# mariadb -u root -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 674
Server version: 11.3.2-MariaDB-1:11.3.2+maria~ubu2204 mariadb.org binary distribution
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> exit
Bye
perfectra1n commented
I went ahead and changed the root password to not include a !, and now the logs look like the following when constantly crashing:
Full Log of a Pod with crash after changing root password
{"level":"info","ts":1713901112.128137,"msg":"Starting agent"}
{"level":"info","ts":1713901112.13175,"logger":"server","msg":"server listening","addr":":5555"}
2024/04/23 19:39:13 "DELETE http://mariadb-gnew-1.mariadb-gnew-internal.mariadb.svc.newcluster.local:5555/api/bootstrap HTTP/1.1" from 10.233.69.21:44386 - 404 41B in 11.645103ms
{"level":"error","ts":1713901162.1703997,"logger":"handler.probe.readiness","msg":"error getting MariaDB","error":"Get \"https://10.233.0.1:443/apis/k8s.mariadb.com/v1alpha1/namespaces/mariadb/mariadbs/mariadb-gnew\": context deadline exceeded","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:96\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901221.1846123,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901221.1885583,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901226.184601,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901226.2316885,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901231.182878,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901231.186301,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901256.190096,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901256.1904545,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901261.1816254,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901261.1843452,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901266.1811845,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901266.1869998,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901266.2037497,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901291.184306,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901291.1846526,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901296.1825287,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901296.1837292,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901301.1870213,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901301.187811,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
[mysql] 2024/04/23 19:41:42 packets.go:37: read tcp [::1]:35878->[::1]:3306: read: connection reset by peer
{"level":"error","ts":1713901302.220712,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"dial tcp [::1]:3306: connect: connection refused","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901303.6034021,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"dial tcp [::1]:3306: connect: connection refused","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901326.1832745,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901326.185197,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901331.1809835,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901331.1821089,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901336.1823082,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901336.1838481,"logger":"handler.probe.liveness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Liveness\n\t/app/pkg/galera/agent/handler/probe.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901336.199937,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"Error 1045: Access denied for user 'root'@'localhost' (using password: YES)","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1713901339.0055046,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"dial tcp [::1]:3306: connect: connection refused","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
2024-04-23 19:48:02+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.3.2+maria~ubu2204 started.
2024-04-23 19:48:02+00:00 [Warn] [Entrypoint]: /sys/fs/cgroup///memory.pressure not writable, functionality unavailable to MariaDB
2024-04-23 19:48:02+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2024-04-23 19:48:02+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.3.2+maria~ubu2204 started.
2024-04-23 19:48:03+00:00 [Note] [Entrypoint]: MariaDB upgrade information missing, assuming required
2024-04-23 19:48:03+00:00 [Note] [Entrypoint]: MariaDB upgrade (mariadb-upgrade or creating healthcheck users) required, but skipped due to $MARIADB_AUTO_UPGRADE setting
2024-04-23 19:48:03 0 [Note] Starting MariaDB 11.3.2-MariaDB-1:11.3.2+maria~ubu2204 source revision 068a6819eb63bcb01fdfa037c9bf3bf63c33ee42 as process 1
2024-04-23 19:48:03 0 [Note] WSREP: Loading provider /usr/lib/galera/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1
2024-04-23 19:48:03 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
2024-04-23 19:48:03 0 [Note] WSREP: wsrep_load(): Galera 26.4.16(r7dce5149) by Codership Oy <info@codership.com> loaded successfully.
2024-04-23 19:48:03 0 [Note] WSREP: Initializing allowlist service v1
2024-04-23 19:48:03 0 [Note] WSREP: Initializing event service v1
2024-04-23 19:48:03 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.
2024-04-23 19:48:03 0 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 0
2024-04-23 19:48:03 0 [Note] WSREP: GCache DEBUG: opened preamble:
Version: 2
UUID: 003b272f-01a9-11ef-beba-026f3aace78f
Seqno: -1 - -1
Offset: -1
Synced: 0
2024-04-23 19:48:03 0 [Note] WSREP: Recovering GCache ring buffer: version: 2, UUID: 003b272f-01a9-11ef-beba-026f3aace78f, offset: -1
2024-04-23 19:48:03 0 [Note] WSREP: GCache::RingBuffer initial scan... 0.0% ( 0/134217752 bytes) complete.
2024-04-23 19:48:03 0 [Note] WSREP: GCache::RingBuffer initial scan...100.0% (134217752/134217752 bytes) complete.
2024-04-23 19:48:03 0 [Note] WSREP: Recovering GCache ring buffer: found gapless sequence 675-1409
2024-04-23 19:48:03 0 [Note] WSREP: GCache::RingBuffer unused buffers scan... 0.0% ( 0/132302552 bytes) complete.
2024-04-23 19:48:03 0 [Note] WSREP: Recovering GCache ring buffer: found 7/742 locked buffers
2024-04-23 19:48:03 0 [Note] WSREP: Recovering GCache ring buffer: free space: 1916680/134217728
2024-04-23 19:48:03 0 [Note] WSREP: GCache::RingBuffer unused buffers scan...100.0% (132302552/132302552 bytes) complete.
2024-04-23 19:48:03 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.233.91.138; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.keep_plaintext_size = 128M; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0
2024-04-23 19:48:03 0 [Note] WSREP: Start replication
2024-04-23 19:48:03 0 [Note] WSREP: Connecting with bootstrap option: 0
2024-04-23 19:48:03 0 [Note] WSREP: Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1
2024-04-23 19:48:03 0 [Note] WSREP: protonet asio version 0
2024-04-23 19:48:03 0 [Note] WSREP: Using CRC-32C for message checksums.
2024-04-23 19:48:03 0 [Note] WSREP: backend: asio
2024-04-23 19:48:03 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
2024-04-23 19:48:03 0 [Note] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
2024-04-23 19:48:03 0 [Note] WSREP: restore pc from disk failed
2024-04-23 19:48:03 0 [Note] WSREP: GMCast version 0
2024-04-23 19:48:03 0 [Note] WSREP: (65c7b769-a4b9, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
2024-04-23 19:48:03 0 [Note] WSREP: (65c7b769-a4b9, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
2024-04-23 19:48:03 0 [Note] WSREP: EVS version 1
2024-04-23 19:48:03 0 [Note] WSREP: gcomm: connecting to group 'mariadb-operator', peer 'mariadb-gnew-0.mariadb-gnew-internal.mariadb.svc.newcluster.local:,mariadb-gnew-1.mariadb-gnew-internal.mariadb.svc.newcluster.local:,mariadb-gnew-2.mariadb-gnew-internal.mariadb.svc.newcluster.local:'
2024-04-23 19:48:03 0 [Note] WSREP: (65c7b769-a4b9, 'tcp://0.0.0.0:4567') Found matching local endpoint for a connection, blacklisting address tcp://10.233.91.138:4567
2024-04-23 19:48:03 0 [Note] WSREP: (65c7b769-a4b9, 'tcp://0.0.0.0:4567') connection established to 1f792853-896d tcp://10.233.101.184:4567
2024-04-23 19:48:03 0 [Note] WSREP: (65c7b769-a4b9, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2024-04-23 19:48:04 0 [Note] WSREP: EVS version upgrade 0 -> 1
2024-04-23 19:48:04 0 [Note] WSREP: declaring 1f792853-896d at tcp://10.233.101.184:4567 stable
2024-04-23 19:48:04 0 [Note] WSREP: PC protocol upgrade 0 -> 1
2024-04-23 19:48:04 0 [Note] WSREP: Node 1f792853-896d state prim
2024-04-23 19:48:04 0 [Note] WSREP: view(view_id(PRIM,1f792853-896d,22) memb {
1f792853-896d,0
65c7b769-a4b9,0
} joined {
} left {
} partitioned {
})
2024-04-23 19:48:04 0 [Note] WSREP: save pc into disk
2024-04-23 19:48:04 0 [Note] WSREP: discarding pending addr without UUID: tcp://10.233.69.10:4567
2024-04-23 19:48:04 0 [Note] WSREP: gcomm: connected
2024-04-23 19:48:04 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
2024-04-23 19:48:04 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
2024-04-23 19:48:04 0 [Note] WSREP: Opened channel 'mariadb-operator'
2024-04-23 19:48:04 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
2024-04-23 19:48:04 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
2024-04-23 19:48:04 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 66165298-01aa-11ef-91b2-43347ae01c3b
2024-04-23 19:48:04 0 [Note] WSREP: STATE EXCHANGE: got state msg: 66165298-01aa-11ef-91b2-43347ae01c3b from 0 (mariadb-gnew-2)
2024-04-23 19:48:04 0 [Note] WSREP: Initializing config service v1
2024-04-23 19:48:04 1 [Note] WSREP: Starting rollbacker thread 1
2024-04-23 19:48:04 2 [Note] WSREP: Starting applier thread 2
2024-04-23 19:48:04 0 [Note] WSREP: STATE EXCHANGE: got state msg: 66165298-01aa-11ef-91b2-43347ae01c3b from 1 (mariadb-gnew-1)
2024-04-23 19:48:04 0 [Note] WSREP: Quorum results:
version = 6,
component = PRIMARY,
conf_id = 20,
members = 1/2 (joined/total),
act_id = 1416,
last_appl. = 1284,
protocols = 2/10/4 (gcs/repl/appl),
vote policy= 0,
group UUID = 003b272f-01a9-11ef-beba-026f3aace78f
2024-04-23 19:48:04 0 [Note] WSREP: Flow-control interval: [23, 23]
2024-04-23 19:48:04 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 1417)
2024-04-23 19:48:04 0 [Note] WSREP: Deinitializing config service v1
2024-04-23 19:48:04 2 [Note] WSREP: ####### processing CC 1417, local, ordered
2024-04-23 19:48:04 2 [Note] WSREP: Process first view: 003b272f-01a9-11ef-beba-026f3aace78f my uuid: 65c7b769-01aa-11ef-a4b9-4fcbab8908cb
2024-04-23 19:48:04 2 [Note] WSREP: Server mariadb-gnew-1 connected to cluster at position 003b272f-01a9-11ef-beba-026f3aace78f:1417 with ID 65c7b769-01aa-11ef-a4b9-4fcbab8908cb
2024-04-23 19:48:04 2 [Note] WSREP: Server status change disconnected -> connected
2024-04-23 19:48:04 2 [Note] WSREP: ####### My UUID: 65c7b769-01aa-11ef-a4b9-4fcbab8908cb
2024-04-23 19:48:04 2 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes
2024-04-23 19:48:04 0 [Note] WSREP: Service thread queue flushed.
2024-04-23 19:48:04 2 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
2024-04-23 19:48:04 2 [Note] WSREP: State transfer required:
Group state: 003b272f-01a9-11ef-beba-026f3aace78f:1417
Local state: 00000000-0000-0000-0000-000000000000:-1
2024-04-23 19:48:04 2 [Note] WSREP: Server status change connected -> joiner
2024-04-23 19:48:04 0 [Note] WSREP: Joiner monitor thread started to monitor
2024-04-23 19:48:04 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.233.91.138' --datadir '/var/lib/mysql/' --parent 1 --progress 0'
WSREP_SST: [INFO] mariabackup SST started on joiner (20240423 19:48:04.639)
WSREP_SST: [INFO] SSL configuration: CA='', CAPATH='', CERT='', KEY='', MODE='DISABLED', encrypt='0' (20240423 19:48:04.791)
WSREP_SST: [INFO] Progress reporting tool pv not found in path: /usr//bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/sbin:/usr/bin:/sbin:/bin (20240423 19:48:05.150)
WSREP_SST: [INFO] Disabling all progress/rate-limiting (20240423 19:48:05.154)
WSREP_SST: [INFO] Streaming with mbstream (20240423 19:48:05.187)
WSREP_SST: [INFO] Using socat as streamer (20240423 19:48:05.191)
WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql/sst_in_progress (20240423 19:48:05.197)
WSREP_SST: [INFO] previous SST is not completed, waiting for it to exit (20240423 19:48:05.252)
WSREP_SST: [INFO] previous SST is not completed, waiting for it to exit (20240423 19:48:06.283)
2024-04-23 19:48:07 0 [Note] WSREP: (65c7b769-a4b9, 'tcp://0.0.0.0:4567') turning message relay requesting off
WSREP_SST: [INFO] previous SST is not completed, waiting for it to exit (20240423 19:48:07.315)
WSREP_SST: [INFO] previous SST is not completed, waiting for it to exit (20240423 19:48:08.350)
WSREP_SST: [INFO] previous SST is not completed, waiting for it to exit (20240423 19:48:09.381)
WSREP_SST: [INFO] previous SST is not completed, waiting for it to exit (20240423 19:48:10.423)
WSREP_SST: [INFO] previous SST is not completed, waiting for it to exit (20240423 19:48:11.458)
WSREP_SST: [INFO] previous SST is not completed, waiting for it to exit (20240423 19:48:12.500)
WSREP_SST: [INFO] previous SST is not completed, waiting for it to exit (20240423 19:48:13.536)
WSREP_SST: [INFO] previous SST is not completed, waiting for it to exit (20240423 19:48:14.574)
WSREP_SST: [ERROR] previous SST script still running. (20240423 19:48:14.584)
2024-04-23 19:48:14 0 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_mariabackup --role 'joiner' --address '10.233.91.138' --datadir '/var/lib/mysql/' --parent 1 --progress 0
Read: '(null)'
2024-04-23 19:48:14 0 [ERROR] WSREP: Process completed with error: wsrep_sst_mariabackup --role 'joiner' --address '10.233.91.138' --datadir '/var/lib/mysql/' --parent 1 --progress 0: 114 (Operation already in progress)
2024-04-23 19:48:14 2 [ERROR] WSREP: Failed to prepare for 'mariabackup' SST. Unrecoverable.
2024-04-23 19:48:14 2 [ERROR] WSREP: SST request callback failed. This is unrecoverable, restart required.
2024-04-23 19:48:14 2 [Note] WSREP: ReplicatorSMM::abort()
2024-04-23 19:48:14 2 [Note] WSREP: Closing send monitor...
2024-04-23 19:48:14 2 [Note] WSREP: Closed send monitor.
2024-04-23 19:48:14 2 [Note] WSREP: gcomm: terminating thread
2024-04-23 19:48:14 2 [Note] WSREP: gcomm: joining thread
2024-04-23 19:48:14 2 [Note] WSREP: gcomm: closing backend
2024-04-23 19:48:15 2 [Note] WSREP: view(view_id(NON_PRIM,1f792853-896d,22) memb {
65c7b769-a4b9,0
} joined {
} left {
} partitioned {
1f792853-896d,0
})
2024-04-23 19:48:15 2 [Note] WSREP: PC protocol downgrade 1 -> 0
2024-04-23 19:48:15 2 [Note] WSREP: view((empty))
2024-04-23 19:48:15 2 [Note] WSREP: gcomm: closed
2024-04-23 19:48:15 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2024-04-23 19:48:15 0 [Note] WSREP: Flow-control interval: [16, 16]
2024-04-23 19:48:15 0 [Note] WSREP: Received NON-PRIMARY.
2024-04-23 19:48:15 0 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 1417)
2024-04-23 19:48:15 0 [Note] WSREP: New SELF-LEAVE.
2024-04-23 19:48:15 0 [Note] WSREP: Flow-control interval: [0, 0]
2024-04-23 19:48:15 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2024-04-23 19:48:15 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 1417)
2024-04-23 19:48:15 0 [Note] WSREP: RECV thread exiting 0: Success
2024-04-23 19:48:15 2 [Note] WSREP: recv_thread() joined.
2024-04-23 19:48:15 2 [Note] WSREP: Closing replication queue.
2024-04-23 19:48:15 2 [Note] WSREP: Closing slave action queue.
2024-04-23 19:48:15 2 [Note] WSREP: mariadbd: Terminated.
240423 19:48:15 [ERROR] mysqld got signal 11 ;
Sorry, we probably made a mistake, and this is a bug.
Your assistance in bug reporting will enable us to fix this for the next release.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 11.3.2-MariaDB-1:11.3.2+maria~ubu2204 source revision: 068a6819eb63bcb01fdfa037c9bf3bf63c33ee42
key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=3
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 336992 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7f6f5c000c68
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f6f81425c68 thread_stack 0x49000
Printing to addr2line failed
mariadbd(my_print_stacktrace+0x32)[0x55a1a86358a2]
mariadbd(handle_fatal_signal+0x478)[0x55a1a8106488]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f6f83bef520]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x178)[0x7f6f83bd5898]
/usr/lib/galera/libgalera_smm.so(+0x157602)[0x7f6f83671602]
/usr/lib/galera/libgalera_smm.so(+0x700e1)[0x7f6f8358a0e1]
/usr/lib/galera/libgalera_smm.so(+0x6cc94)[0x7f6f83586c94]
/usr/lib/galera/libgalera_smm.so(+0x8b311)[0x7f6f835a5311]
/usr/lib/galera/libgalera_smm.so(+0x604a0)[0x7f6f8357a4a0]
/usr/lib/galera/libgalera_smm.so(+0x48261)[0x7f6f83562261]
mariadbd(_ZN5wsrep18wsrep_provider_v2611run_applierEPNS_21high_priority_serviceE+0x12)[0x55a1a86f5592]
mariadbd(+0xd93e31)[0x55a1a83c5e31]
mariadbd(_Z15start_wsrep_THDPv+0x26b)[0x55a1a83b3a7b]
mariadbd(+0xd05f86)[0x55a1a8337f86]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f6f83c41ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f6f83cd3850]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0): (null)
Connection ID (thread ID): 2
Status: NOT_KILLED
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off,hash_join_cardinality=on,cset_narrowing=off,sargable_casefold=on
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/ contains
information that should help you find out what is causing the crash.
We think the query pointer is invalid, but we will try to print it anyway.
Query:
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 0 bytes
Max resident set unlimited unlimited bytes
Max processes unlimited unlimited processes
Max open files 65535 65535 files
Max locked memory unlimited unlimited bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 192953 192953 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Core pattern: core
Kernel version: Linux version 5.10.0-28-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.209-2 (2024-01-31)
mmontes11 commented
Hey there @perfectra1n !
There is a known issue with the mysql.global_priv table and Galera:
This has been resolved via #545 but not released yet.
Also FYI, there is another limitation with the LOCK TABLES statements and Galera:
It has been resolved in #462
I believe this should cover your problems, let me know otherwise and stay tuned for v0.0.28, which is aimed to be released this week!
perfectra1n commented
Oh hey @mmontes11 thanks for getting back so quickly, this was seriously making me lose my mind lol. Are you aware of any workarounds that I'm able to use to at least get my cluster back up? I'll keep trying to brainstorm...
mmontes11 commented
The propper solution will be upgrading to v0.0.28 when released, something which will ignore the mysql.global_priv table when taking Backups.
As a workaround, but I do not recommend this in a production environment, you could manually delete mysql.global_priv related statements in the backup and then restore it. This will be automatically handled in v0.0.28
perfectra1n commented
Thanks for responding so quickly, I didn't see those issues when searching!
perfectra1n commented
Well I took the one good backup that I do have, and restored it to a MariaDB Docker container, and then ran the following command to generate a "good" backup:
mariadb-dump --single-transaction --events --routines --all-databases --skip-add-locks --ignore-table=mysql.global_priv -p > backup.sqlHowever, it looks like the "new to me" User and Grant objects aren't replicated across all 3 nodes.
Sometimes I'm able to run the SQL:
SELECT User FROM mysql.user;
And other times I'm not (which I believe are the nodes that have the correct User and Grants, so this is the expected behavior):
Which results in the following uptime notifications:
Any ideas of how I would "sync" the users across all 3 nodes?
perfectra1n commented
I already have User and Grant resources such as:
apiVersion: k8s.mariadb.com/v1alpha1
kind: User
metadata:
name: bookstack
namespace: mariadb
spec:
# If you want the user to be created with a different name than the resource name
# name: user-custom
mariaDbRef:
name: mariadb-fixed
passwordSecretKeyRef:
name: bookstack-creds
key: password
# This field is immutable and defaults to 10
maxUserConnections: 50
host: "%"
requeueInterval: 30s
retryInterval: 5sand
apiVersion: k8s.mariadb.com/v1alpha1
kind: Grant
metadata:
name: bookstack-grant
namespace: mariadb
spec:
mariaDbRef:
name: mariadb-fixed
privileges:
- "ALL PRIVILEGES"
database: "bookstackdb"
table: "*"
username: bookstack
grantOption: true
host: "%"
requeueInterval: 30s
retryInterval: 5s
But sometimes the users work, and other times they don't (I'm assuming because of the Users being different across nodes)
perfectra1n commented
Okay, I got it to work. I deleted both the Galera and Storage PVCs, then killed the pod where the User and Grants weren't working.
I used the following command locally within the MariaDB pod to see if it wasn't "cooperating":
SELECT User FROM mysql.user;When the pod/DB node rejoined, it then worked as expected.
perfectra1n commented
Unfortunately the issues from these two comments, this and this keep happening. Sometimes the User and Grant mariadb-operator types work (where mysql.users is just a View), and other times they don't, leading to failure to authenticate.
Scaling up doesn't work nor recreating pod's PVCs, errors like the following keep happening:
New node in Galera cluster just crashing
00000000-0000-0000-0000-000000000000:1999 with ID c6fb778a-0294-11ef-9cc5-b705eaf53c1b
2024-04-24 23:45:49 2 [Note] WSREP: Server status change disconnected -> connected
2024-04-24 23:45:49 2 [Note] WSREP: ####### My UUID: c6fb778a-0294-11ef-9cc5-b705eaf53c1b
2024-04-24 23:45:49 2 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes
2024-04-24 23:45:49 0 [Note] WSREP: Service thread queue flushed.
2024-04-24 23:45:49 2 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
2024-04-24 23:45:49 2 [Note] WSREP: State transfer required:
Group state: 00000000-0000-0000-0000-000000000000:1999
Local state: 00000000-0000-0000-0000-000000000000:-1
2024-04-24 23:45:49 2 [Note] WSREP: Server status change connected -> joiner
2024-04-24 23:45:49 0 [Note] WSREP: Joiner monitor thread started to monitor
2024-04-24 23:45:49 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.233.91.176' --datadir '/var/lib/mysql/' --parent 1 --progress 0'
WSREP_SST: [INFO] mariabackup SST started on joiner (20240424 23:45:49.972)
WSREP_SST: [INFO] SSL configuration: CA='', CAPATH='', CERT='', KEY='', MODE='DISABLED', encrypt='0' (20240424 23:45:50.152)
WSREP_SST: [INFO] Progress reporting tool pv not found in path: /usr//bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/sbin:/usr/bin:/sbin:/bin (20240424 23:45:50.529)
WSREP_SST: [INFO] Disabling all progress/rate-limiting (20240424 23:45:50.538)
WSREP_SST: [INFO] Streaming with mbstream (20240424 23:45:50.607)
WSREP_SST: [INFO] Using socat as streamer (20240424 23:45:50.616)
WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | '/usr//bin/mbstream' -x; RC=( ${PIPESTATUS[@]} ) (20240424 23:45:50.784)
2024-04-24 23:45:51 2 [Note] WSREP: ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 1999, STRv: 3
2024-04-24 23:45:51 2 [Note] WSREP: IST receiver addr using tcp://10.233.91.176:4568
2024-04-24 23:45:51 2 [Note] WSREP: Prepared IST receiver for 0-1999, listening at: tcp://10.233.91.176:4568
2024-04-24 23:45:51 0 [Note] WSREP: Member 2.0 (mariadb-fixed-4) requested state transfer from '*any*'. Selected 0.0 (mariadb-fixed-1)(SYNCED) as donor.
2024-04-24 23:45:51 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 2000)
2024-04-24 23:45:51 2 [Note] WSREP: Requesting state transfer: success, donor: 0
WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | '/usr//bin/mbstream' -x; RC=( ${PIPESTATUS[@]} ) (20240424 23:45:51.404)
WSREP_SST: [INFO] Proceeding with SST (20240424 23:45:51.404)
WSREP_SST: [INFO] Cleaning the existing datadir and innodb-data/log directories (20240424 23:45:51.500)
WSREP_SST: [INFO] Waiting for SST streaming to complete! (20240424 23:45:51.562)
2024-04-24 23:45:52 0 [Note] WSREP: (c6fb778a-9cc5, 'tcp://0.0.0.0:4567') turning message relay requesting off
{"level":"info","ts":1714002365.3846986,"logger":"handler.probe.liveness","msg":"Galera not ready. Returning OK to facilitate recovery"}
{"level":"info","ts":1714002365.3864858,"logger":"handler.probe.readiness","msg":"Galera not ready. Returning OK to facilitate recovery"}
{"level":"info","ts":1714002370.355669,"logger":"handler.probe.liveness","msg":"Galera not ready. Returning OK to facilitate recovery"}
2024-04-24 23:46:11 0 [Note] WSREP: 0.0 (mariadb-fixed-1): State transfer to 2.0 (mariadb-fixed-4) complete.
2024-04-24 23:46:11 0 [Note] WSREP: Member 0.0 (mariadb-fixed-1) synced with group.
WSREP_SST: [INFO] Preparing the backup at /var/lib/mysql/.sst (20240424 23:46:11.201)
WSREP_SST: [INFO] Evaluating /usr//bin/mariadb-backup --prepare --no-version-check --target-dir='/var/lib/mysql/.sst' --datadir='/var/lib/mysql/.sst' > '/var/lib/mysql/mariabackup.prepare.log' 2>&1 (20240424 23:46:11.215)
{"level":"error","ts":1714002371.3374262,"logger":"handler.probe.readiness","msg":"error getting MariaDB","error":"Get \"https://10.233.0.1:443/apis/k8s.mariadb.com/v1alpha1/namespaces/mariadb/mariadbs/mariadb-fixed\": context deadline exceeded","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:96\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1714002371.3395054,"logger":"handler.probe.readiness","msg":"error getting SQL client","error":"dial tcp [::1]:3306: connect: connection refused","stacktrace":"github.com/mariadb-operator/mariadb-operator/pkg/galera/agent/handler.(*Probe).Readiness\n\t/app/pkg/galera/agent/handler/probe.go:117\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.Recoverer.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/recoverer.go:43\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.(*Compressor).Handler-fm.(*Compressor).Handler.func1\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/compress.go:213\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\t/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
WSREP_SST: [INFO] Moving the backup to /var/lib/mysql (20240424 23:46:12.619)
WSREP_SST: [INFO] Evaluating /usr//bin/mariadb-backup --move-back --no-version-check --galera-info --force-non-empty-directories --target-dir='/var/lib/mysql/.sst' --datadir='/var/lib/mysql' > '/var/lib/mysql/mariabackup.move.log' 2>&1 (20240424 23:46:12.638)
WSREP_SST: [INFO] Move successful, removing /var/lib/mysql/.sst (20240424 23:46:12.788)
WSREP_SST: [INFO] Galera co-ords from donor: :2019 0 (20240424 23:46:12.810)
WSREP_SST: [INFO] Total time on joiner: 0 seconds (20240424 23:46:12.818)
WSREP_SST: [INFO] mariabackup SST completed on joiner (20240424 23:46:12.827)
WSREP_SST: [INFO] Removing the sst_in_progress file (20240424 23:46:12.841)
WSREP_SST: [INFO] Cleaning up temporary directories (20240424 23:46:12.854)
2024-04-24 23:46:12 0 [ERROR] WSREP: Failed to parse uuid:seqno pair: ':2019'
2024-04-24 23:46:12 3 [Note] WSREP: SST received
2024-04-24 23:46:12 3 [Note] WSREP: SST received: 00000000-0000-0000-0000-000000000000:-1
2024-04-24 23:46:12 3 [Note] WSREP: SST succeeded for position 00000000-0000-0000-0000-000000000000:-1
2024-04-24 23:46:12 2 [Note] WSREP: Installed new state from SST: 00000000-0000-0000-0000-000000000000:-1
2024-04-24 23:46:12 0 [Note] WSREP: Joiner monitor thread ended with total time 23 sec
2024-04-24 23:46:12 2 [Note] WSREP: Cert. index preload up to -1
2024-04-24 23:46:12 0 [Note] WSREP: ####### IST applying starts with 0
2024-04-24 23:46:12 0 [ERROR] WSREP: IST started with wrong seqno: 1937, expected <= 0
2024-04-24 23:46:12 0 [ERROR] WSREP: IST didn't contain all write sets, expected last: 1999 last received: -1
2024-04-24 23:46:12 2 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 71 (Protocol error)
at ./galera/src/replicator_smm.hpp:pop_front():336. Null event.
2024-04-24 23:46:12 2 [Note] WSREP: Closing send monitor...
2024-04-24 23:46:12 2 [Note] WSREP: Closed send monitor.
2024-04-24 23:46:12 2 [Note] WSREP: gcomm: terminating thread
2024-04-24 23:46:12 2 [Note] WSREP: gcomm: joining thread
2024-04-24 23:46:12 2 [Note] WSREP: gcomm: closing backend
2024-04-24 23:46:12 2 [Note] WSREP: view(view_id(NON_PRIM,1b3a5cbf-9b68,32) memb {
c6fb778a-9cc5,0
} joined {
} left {
} partitioned {
1b3a5cbf-9b68,0
264cedbf-91f8,0
})
2024-04-24 23:46:12 2 [Note] WSREP: PC protocol downgrade 1 -> 0
2024-04-24 23:46:12 2 [Note] WSREP: view((empty))
2024-04-24 23:46:12 2 [Note] WSREP: gcomm: closed
2024-04-24 23:46:12 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2024-04-24 23:46:12 0 [Note] WSREP: Flow-control interval: [16, 16]
2024-04-24 23:46:12 0 [Note] WSREP: Received NON-PRIMARY.
2024-04-24 23:46:12 0 [Note] WSREP: Shifting JOINER -> OPEN (TO: 2019)
2024-04-24 23:46:12 0 [Note] WSREP: New SELF-LEAVE.
2024-04-24 23:46:12 0 [Note] WSREP: Flow-control interval: [0, 0]
2024-04-24 23:46:12 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2024-04-24 23:46:12 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 2019)
2024-04-24 23:46:12 0 [Note] WSREP: RECV thread exiting 0: Success
2024-04-24 23:46:12 2 [Note] WSREP: recv_thread() joined.
2024-04-24 23:46:12 2 [Note] WSREP: Closing replication queue.
2024-04-24 23:46:12 2 [Note] WSREP: Closing slave action queue.
2024-04-24 23:46:12 2 [Note] WSREP: IST received: 00000000-0000-0000-0000-000000000000:-1
2024-04-24 23:46:12 2 [ERROR] WSREP: ./gcs/src/gcs.cpp:s_join():965: Sending JOIN failed: -103 (Software caused connection abort).
2024-04-24 23:46:12 2 [Warning] WSREP: Failed to JOIN the cluster after SST gcs_join(00000000-0000-0000-0000-000000000000:-1) failed: 103 (Software caused connection abort)
at ./galera/src/galera_gcs.hpp:join():231
2024-04-24 23:46:12 2 [Note] WSREP: Lowest cert index boundary for CC from sst: -1
2024-04-24 23:46:12 2 [Note] WSREP: Min available from gcache for CC from sst: 1937
2024-04-24 23:46:12 2 [Warning] WSREP: trx protocol version: 5 does not match certification protocol version: -1
{"level":"info","ts":1714002375.3575919,"logger":"handler.probe.readiness","msg":"Galera not ready. Returning OK to facilitate recovery"}
{"level":"info","ts":1714002375.3581927,"logger":"handler.probe.liveness","msg":"Galera not ready. Returning OK to facilitate recovery"}
{"level":"info","ts":1714002380.3503997,"logger":"handler.probe.liveness","msg":"Galera not ready. Returning OK to facilitate recovery"}
Not sure if you've seen this as well @mmontes11, but might be something worth resolving for the new version of the operator as well! I had 3 nodes and went to 5, and this exact issue just keeps playing out across the 2 new nodes.
mmontes11 commented
Hey there @perfectra1n ! Good timing for reporting this, I'm doing e2e testing of the release, very much appreciated your issue reports here.
When a Galera cluster gets upscaled, the new node receives an SST, which will not contain the mysql.global_priv with the data related to the users as this table is backed by the InnoDB engine, more info in #556
However, the docker image understands the MARIADB_ROOT_PASSWORD env, which will setup the root password used by the liveness and readiness probe so I am not sure at this point why it keeps failing. I will test this case with v0.0.28 and take further action if needed.
Regarding the User and Grant resources, they are reconciled against the primary node, but this might not guarantee that it gets propagated for the other nodes. Will test this case with v0.0.28 again. Worst case scenario: we can change the User and Grant reconciliation logic and perform the queries, not only in the primary nodes, but in all nodes when Galera is enabled. This won't generate deadlocks of any kind, since the mysql.global_priv is not handled by Galera.
mmontes11 commented
We are tracking the cluster upscale in a different issue, which is related to a pending SST:
Here we will stick to discuss the behaviour of User and Grant resources with Galera in a backup restoration scenario
mmontes11 commented
@perfectra1n I've managed to reproduce your problem described in #580 (comment)
The Users and Grants are continiously reconciled by the operator, resulting in the mysql.global_priv being populated in all the nodes but... when loging in, you access via the mysql.user view, and for doing so MariaDB uses the mariadb.sys user, which ofc also needs privileges in mysql.global_priv. Since we are omiting mysql.global_priv in backups, the privileges of mariadb.sys are not getting created.
In one of the nodes that you get the following error:
MariaDB [mysql]> select * from user;
ERROR 1356 (HY000): View 'mysql.user' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use themcould you please run the following queries?:
CREATE USER IF NOT EXISTS `mariadb.sys`@`localhost` ACCOUNT LOCK PASSWORD EXPIRE;
GRANT SELECT, UPDATE, DELETE ON `mysql`.`global_priv` TO `mariadb.sys`@`localhost`;Then you should be able to login using the User and Grant resources. Could you please confirm this?
mmontes11 commented
@perfectra1n one more thing. Although I do get an error when querying select * from mysql.user I am still able co connect:
root@mariadb-galera-1:/# mariadb -u root -pMariaDB11!
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 248
Server version: 10.11.7-MariaDB-1:10.11.7+maria~ubu2204 mariadb.org binary distribution
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> select User from mysql.user;
ERROR 1356 (HY000): View 'mysql.user' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them
MariaDB [(none)]> exit
Bye
root@mariadb-galera-1:/# mariadb -u user -pMariaDB11!
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 254
Server version: 10.11.7-MariaDB-1:10.11.7+maria~ubu2204 mariadb.org binary distribution
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]>
Could you confirm this?
mmontes11 commented
Hey there @perfectra1n !
This has been fixed by #597. You should be able to connect after restoring Backups in Galera using credentials defined via User and Grant resources.
Stay tuned for v0.0.28!
perfectra1n commented
Awesome, thank you so much! I scaled my DB down to 1 so that I wouldn't have to mess around with Galera for the moment, since I just needed a working DB :)
perfectra1n commented
Yeah, it just keeps erroring out after restoring at v0.0.28 from S3:
It's just this over, and over again:
Node in Galera cluster just crashing
2024-05-11 00:47:36+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.3.2+maria~ubu2204 started.
2024-05-11 00:47:36+00:00 [Warn] [Entrypoint]: /sys/fs/cgroup///memory.pressure not writable, functionality unavailable to MariaDB
2024-05-11 00:47:36+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2024-05-11 00:47:36+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.3.2+maria~ubu2204 started.
2024-05-11 00:47:37+00:00 [Note] [Entrypoint]: MariaDB upgrade information missing, assuming required
2024-05-11 00:47:37+00:00 [Note] [Entrypoint]: MariaDB upgrade (mariadb-upgrade or creating healthcheck users) required, but skipped due to $MARIADB_AUTO_UPGRADE setting
2024-05-11 0:47:37 0 [Note] Using unique option prefix 'ignore-db-dir' is error-prone and can break in the future. Please use the full name 'ignore_db_dirs' instead.
2024-05-11 0:47:37 0 [Note] Starting MariaDB 11.3.2-MariaDB-1:11.3.2+maria~ubu2204 source revision 068a6819eb63bcb01fdfa037c9bf3bf63c33ee42 as process 1
2024-05-11 0:47:37 0 [Note] WSREP: Loading provider /usr/lib/galera/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1
2024-05-11 0:47:37 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
2024-05-11 0:47:37 0 [Note] WSREP: wsrep_load(): Galera 26.4.16(r7dce5149) by Codership Oy <info@codership.com> loaded successfully.
2024-05-11 0:47:37 0 [Note] WSREP: Initializing allowlist service v1
2024-05-11 0:47:37 0 [Note] WSREP: Initializing event service v1
2024-05-11 0:47:37 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.
2024-05-11 0:47:37 0 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 0
2024-05-11 0:47:37 0 [Note] WSREP: GCache DEBUG: opened preamble:
Version: 2
UUID: 75853597-0f26-11ef-a58f-5a996b7c1ae7
Seqno: -1 - -1
Offset: -1
Synced: 0
2024-05-11 0:47:37 0 [Note] WSREP: Recovering GCache ring buffer: version: 2, UUID: 75853597-0f26-11ef-a58f-5a996b7c1ae7, offset: -1
2024-05-11 0:47:37 0 [Note] WSREP: GCache::RingBuffer initial scan... 0.0% ( 0/134217752 bytes) complete.
2024-05-11 0:47:37 0 [Note] WSREP: GCache::RingBuffer initial scan...100.0% (134217752/134217752 bytes) complete.
2024-05-11 0:47:37 0 [Note] WSREP: Recovering GCache ring buffer: found gapless sequence 365-876
2024-05-11 0:47:37 0 [Note] WSREP: GCache::RingBuffer unused buffers scan... 0.0% ( 0/47971120 bytes) complete.
2024-05-11 0:47:37 0 [Note] WSREP: Recovering GCache ring buffer: found 0/512 locked buffers
2024-05-11 0:47:37 0 [Note] WSREP: Recovering GCache ring buffer: free space: 86246608/134217728
2024-05-11 0:47:37 0 [Note] WSREP: GCache::RingBuffer unused buffers scan...100.0% (47971120/47971120 bytes) complete.
2024-05-11 0:47:37 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.233.101.166; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.keep_plaintext_size = 128M; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_add
2024-05-11 0:47:37 0 [Note] WSREP: Start replication
2024-05-11 0:47:37 0 [Note] WSREP: Connecting with bootstrap option: 0
2024-05-11 0:47:37 0 [Note] WSREP: Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1
2024-05-11 0:47:37 0 [Note] WSREP: protonet asio version 0
2024-05-11 0:47:37 0 [Note] WSREP: Using CRC-32C for message checksums.
2024-05-11 0:47:37 0 [Note] WSREP: backend: asio
2024-05-11 0:47:37 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
2024-05-11 0:47:37 0 [Note] WSREP: restore pc from disk successfully
2024-05-11 0:47:37 0 [Note] WSREP: GMCast version 0
2024-05-11 0:47:37 0 [Note] WSREP: (9cc9001c-a1ff, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
2024-05-11 0:47:37 0 [Note] WSREP: (9cc9001c-a1ff, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
2024-05-11 0:47:37 0 [Note] WSREP: EVS version 1
2024-05-11 0:47:37 0 [Note] WSREP: gcomm: connecting to group 'mariadb-operator', peer 'galera-mariadb-0.galera-mariadb-internal.mariadb.svc.newcluster.local:,galera-mariadb-1.galera-mariadb-internal.mariadb.svc.newcluster.local:,galera-mariadb-2.galera-mariadb-internal.mariadb.svc.newcluster.local:,galera-mariadb-3.galera-mariadb-internal.mariadb.svc.newcluster.local:,galera-mariadb-4.galera-mariadb-internal.mariadb.svc.newcluster.local:'
2024-05-11 0:47:37 0 [Note] WSREP: (9cc9001c-a1ff, 'tcp://0.0.0.0:4567') Found matching local endpoint for a connection, blacklisting address tcp://10.233.101.166:4567
2024-05-11 0:47:37 0 [Note] WSREP: (9cc9001c-a1ff, 'tcp://0.0.0.0:4567') connection established to 09adb773-a266 tcp://10.233.69.31:4567
2024-05-11 0:47:37 0 [Note] WSREP: (9cc9001c-a1ff, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2024-05-11 0:47:37 0 [Note] WSREP: (9cc9001c-a1ff, 'tcp://0.0.0.0:4567') connection established to f02cbcd0-a12d tcp://10.233.69.47:4567
2024-05-11 0:47:37 0 [Note] WSREP: gcomm: connected
2024-05-11 0:47:37 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
2024-05-11 0:47:37 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
2024-05-11 0:47:37 0 [Note] WSREP: Opened channel 'mariadb-operator'
2024-05-11 0:47:37 0 [Note] WSREP: Initializing config service v1
2024-05-11 0:47:37 1 [Note] WSREP: Starting rollbacker thread 1
2024-05-11 0:47:37 2 [Note] WSREP: Starting applier thread 2
2024-05-11 0:47:37 0 [Note] WSREP: Deinitializing config service v1
2024-05-11 0:47:38 0 [Note] WSREP: (9cc9001c-a1ff, 'tcp://0.0.0.0:4567') connection established to 10c7200d-a956 tcp://10.233.94.154:4567
2024-05-11 0:47:42 0 [Note] WSREP: (9cc9001c-a1ff, 'tcp://0.0.0.0:4567') turning message relay requesting off
2024-05-11 0:47:42 0 [Note] WSREP: evs::proto(9cc9001c-a1ff, GATHER, view_id(TRANS,9cc9001c-a1ff,13)) suspecting node: 9cc9001c-a1fe
2024-05-11 0:47:42 0 [Note] WSREP: evs::proto(9cc9001c-a1ff, GATHER, view_id(TRANS,9cc9001c-a1ff,13)) suspected node without join message, declaring inactive
2024-05-11 0:47:45 0 [Note] WSREP: EVS version upgrade 0 -> 1
2024-05-11 0:47:45 0 [Note] WSREP: PC protocol upgrade 0 -> 1
2024-05-11 0:47:45 0 [Warning] WSREP: no nodes coming from prim view, prim not possible
2024-05-11 0:47:45 0 [Note] WSREP: view(view_id(NON_PRIM,9cc9001c-a1ff,14) memb {
9cc9001c-a1ff,0
} joined {
} left {
} partitioned {
})
2024-05-11 0:47:45 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2024-05-11 0:47:45 0 [Note] WSREP: Flow-control interval: [16, 16]
2024-05-11 0:47:45 0 [Note] WSREP: Received NON-PRIMARY.
2024-05-11 0:47:45 2 [Note] WSREP: ================================================
View:
id: 00000000-0000-0000-0000-000000000000:-1
status: non-primary
protocol_version: -1
capabilities:
final: no
own_index: 0
members(1):
0: 9cc9001c-0f27-11ef-a1ff-128da69be7c9, galera-mariadb-2
=================================================
2024-05-11 0:47:45 2 [Note] WSREP: Non-primary view
2024-05-11 0:47:45 2 [Note] WSREP: Server status change disconnected -> connected
2024-05-11 0:47:45 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2024-05-11 0:47:46 0 [Warning] WSREP: Failed to unserialize message. This may be a result of corrupt message, port scanner or another application connecting to group communication port.
2024-05-11 0:47:52 0 [Warning] WSREP: no nodes coming from prim view, prim not possible
2024-05-11 0:47:52 0 [Note] WSREP: view(view_id(NON_PRIM,9cc9001c-a1ff,15) memb {
9cc9001c-a1ff,0
} joined {
} left {
} partitioned {
})
2024-05-11 0:47:52 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2024-05-11 0:47:52 0 [Note] WSREP: Flow-control interval: [16, 16]
2024-05-11 0:47:52 0 [Note] WSREP: Received NON-PRIMARY.
2024-05-11 0:47:52 2 [Note] WSREP: ================================================
View:
id: 00000000-0000-0000-0000-000000000000:-1
status: non-primary
protocol_version: -1
capabilities:
final: no
own_index: 0
members(1):
0: 9cc9001c-0f27-11ef-a1ff-128da69be7c9, galera-mariadb-2
=================================================
2024-05-11 0:47:52 2 [Note] WSREP: Non-primary view
2024-05-11 0:47:52 2 [Note] WSREP: Server status change connected -> connected
2024-05-11 0:47:52 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2024-05-11 0:47:57 0 [Note] WSREP: (9cc9001c-a1ff, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.233.69.31:4567
2024-05-11 0:47:59 0 [Note] WSREP: (9cc9001c-a1ff, 'tcp://0.0.0.0:4567') reconnecting to 09adb773-a266 (tcp://10.233.69.31:4567), attempt 0
2024-05-11 0:47:59 0 [Warning] WSREP: evs::proto(9cc9001c-a1ff, GATHER, view_id(REG,9cc9001c-a1ff,15)) install timer expired
evs::proto(evs::proto(9cc9001c-a1ff, GATHER, view_id(REG,9cc9001c-a1ff,15)), GATHER) {
current_view=view(view_id(REG,9cc9001c-a1ff,15) memb {
9cc9001c-a1ff,0
} joined {
} left {
} partitioned {
}),
input_map=evs::input_map: {aru_seq=0,safe_seq=0,node_index=node: {idx=0,range=[1,0],safe_seq=0} },
fifo_seq=63,
last_sent=0,
known:
09adb773-a266 at tcp://10.233.69.31:4567
{o=1,s=0,i=0,fs=87,jm=
{v=1,t=4,ut=255,o=1,s=0,sr=-1,as=0,f=4,src=09adb773-a266,srcvid=view_id(REG,09adb773-a266,168),insvid=view_id(UNKNOWN,00000000-0000,0),ru=00000000-0000,r=[-1,-1],fs=87,nl=(
09adb773-a266, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,09adb773-a266,168),ss=0,ir=[1,0],}
10c7200d-a956, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,09adb773-a266,168),ss=0,ir=[1,0],}
9cc9001c-a1ff, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,9cc9001c-a1ff,15),ss=0,ir=[1,0],}
f02cbcd0-a12d, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,09adb773-a266,168),ss=0,ir=[1,0],}
)
},
}
10c7200d-a956 at tcp://10.233.94.154:4567
{o=0,s=0,i=0,fs=52,jm=
{v=1,t=4,ut=255,o=1,s=0,sr=-1,as=0,f=4,src=10c7200d-a956,srcvid=view_id(REG,09adb773-a266,168),insvid=view_id(UNKNOWN,00000000-0000,0),ru=00000000-0000,r=[-1,-1],fs=52,nl=(
09adb773-a266, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,09adb773-a266,168),ss=0,ir=[1,0],}
10c7200d-a956, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,09adb773-a266,168),ss=0,ir=[1,0],}
9cc9001c-a1ff, {o=0,s=0,e=0,ls=-1,vid=view_id(REG,00000000-0000,0),ss=-1,ir=[-1,-1],}
f02cbcd0-a12d, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,09adb773-a266,168),ss=0,ir=[1,0],}
)
},
}
9cc9001c-a1ff at
{o=1,s=0,i=0,fs=-1,jm=
{v=1,t=4,ut=255,o=1,s=0,sr=-1,as=0,f=0,src=9cc9001c-a1ff,srcvid=view_id(REG,9cc9001c-a1ff,15),insvid=view_id(UNKNOWN,00000000-0000,0),ru=00000000-0000,r=[-1,-1],fs=63,nl=(
09adb773-a266, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,09adb773-a266,168),ss=0,ir=[1,0],}
10c7200d-a956, {o=0,s=0,e=0,ls=-1,vid=view_id(REG,09adb773-a266,168),ss=0,ir=[1,0],}
9cc9001c-a1ff, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,9cc9001c-a1ff,15),ss=0,ir=[1,0],}
f02cbcd0-a12d, {o=0,s=1,e=0,ls=-1,vid=view_id(REG,00000000-0000,0),ss=-1,ir=[-1,-1],}
)
},
}
f02cbcd0-a12d at tcp://10.233.69.47:4567
{o=0,s=1,i=0,fs=58932,}
}
2024-05-11 0:47:59 0 [Note] WSREP: no install message received
2024-05-11 0:47:59 0 [Warning] WSREP: no nodes coming from prim view, prim not possible
2024-05-11 0:47:59 0 [Note] WSREP: view(view_id(NON_PRIM,9cc9001c-a1ff,16) memb {
9cc9001c-a1ff,0
} joined {
} left {
} partitioned {
})
2024-05-11 0:47:59 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2024-05-11 0:47:59 0 [Note] WSREP: Flow-control interval: [16, 16]
2024-05-11 0:47:59 0 [Note] WSREP: Received NON-PRIMARY.
2024-05-11 0:47:59 2 [Note] WSREP: ================================================
View:
id: 00000000-0000-0000-0000-000000000000:-1
status: non-primary
protocol_version: -1
capabilities:
final: no
own_index: 0
members(1):
0: 9cc9001c-0f27-11ef-a1ff-128da69be7c9, galera-mariadb-2
=================================================
2024-05-11 0:47:59 2 [Note] WSREP: Non-primary view
2024-05-11 0:47:59 2 [Note] WSREP: Server status change connected -> connected
2024-05-11 0:47:59 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2024-05-11 0:48:05 0 [Warning] WSREP: no nodes coming from prim view, prim not possible
2024-05-11 0:48:05 0 [Note] WSREP: view(view_id(NON_PRIM,9cc9001c-a1ff,17) memb {
9cc9001c-a1ff,0
} joined {
} left {
} partitioned {
})
2024-05-11 0:48:05 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2024-05-11 0:48:05 0 [Note] WSREP: Flow-control interval: [16, 16]
2024-05-11 0:48:05 0 [Note] WSREP: Received NON-PRIMARY.
2024-05-11 0:48:05 2 [Note] WSREP: ================================================
View:
id: 00000000-0000-0000-0000-000000000000:-1
status: non-primary
protocol_version: -1
capabilities:
final: no
own_index: 0
members(1):
0: 9cc9001c-0f27-11ef-a1ff-128da69be7c9, galera-mariadb-2
=================================================
2024-05-11 0:48:05 2 [Note] WSREP: Non-primary view
2024-05-11 0:48:05 2 [Note] WSREP: Server status change connected -> connected
2024-05-11 0:48:05 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2024-05-11 0:48:11 0 [Note] WSREP: (9cc9001c-a1ff, 'tcp://0.0.0.0:4567') reconnecting to 10c7200d-a956 (tcp://10.233.94.154:4567), attempt 0
2024-05-11 0:48:13 0 [Warning] WSREP: evs::proto(9cc9001c-a1ff, GATHER, view_id(REG,9cc9001c-a1ff,17)) install timer expired
evs::proto(evs::proto(9cc9001c-a1ff, GATHER, view_id(REG,9cc9001c-a1ff,17)), GATHER) {
current_view=view(view_id(REG,9cc9001c-a1ff,17) memb {
9cc9001c-a1ff,0
} joined {
} left {
} partitioned {
}),
input_map=evs::input_map: {aru_seq=0,safe_seq=0,node_index=node: {idx=0,range=[1,0],safe_seq=0} },
fifo_seq=99,
last_sent=0,
known:
10c7200d-a956 at tcp://10.233.94.154:4567
{o=1,s=0,i=0,fs=88,jm=
{v=1,t=4,ut=255,o=1,s=0,sr=-1,as=0,f=4,src=10c7200d-a956,srcvid=view_id(REG,10c7200d-a956,170),insvid=view_id(UNKNOWN,00000000-0000,0),ru=00000000-0000,r=[-1,-1],fs=88,nl=(
10c7200d-a956, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,10c7200d-a956,170),ss=0,ir=[1,0],}
9cc9001c-a1ff, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,9cc9001c-a1ff,17),ss=0,ir=[1,0],}
f02cbcd0-a12d, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,10c7200d-a956,170),ss=0,ir=[1,0],}
)
},
}
9cc9001c-a1ff at
{o=1,s=0,i=0,fs=-1,jm=
{v=1,t=4,ut=255,o=1,s=0,sr=-1,as=0,f=0,src=9cc9001c-a1ff,srcvid=view_id(REG,9cc9001c-a1ff,17),insvid=view_id(UNKNOWN,00000000-0000,0),ru=00000000-0000,r=[-1,-1],fs=99,nl=(
10c7200d-a956, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,10c7200d-a956,170),ss=0,ir=[1,0],}
9cc9001c-a1ff, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,9cc9001c-a1ff,17),ss=0,ir=[1,0],}
f02cbcd0-a12d, {o=0,s=1,e=0,ls=-1,vid=view_id(REG,00000000-0000,0),ss=-1,ir=[-1,-1],}
)
},
}
f02cbcd0-a12d at tcp://10.233.69.47:4567
{o=0,s=1,i=0,fs=58962,}
}
2024-05-11 0:48:13 0 [Note] WSREP: no install message received
2024-05-11 0:48:13 0 [Warning] WSREP: no nodes coming from prim view, prim not possible
2024-05-11 0:48:13 0 [Note] WSREP: view(view_id(NON_PRIM,9cc9001c-a1ff,18) memb {
9cc9001c-a1ff,0
} joined {
} left {
} partitioned {
})
2024-05-11 0:48:13 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2024-05-11 0:48:13 0 [Note] WSREP: Flow-control interval: [16, 16]
2024-05-11 0:48:13 0 [Note] WSREP: Received NON-PRIMARY.
2024-05-11 0:48:13 2 [Note] WSREP: ================================================
View:
id: 00000000-0000-0000-0000-000000000000:-1
status: non-primary
protocol_version: -1
capabilities:
final: no
own_index: 0
members(1):
0: 9cc9001c-0f27-11ef-a1ff-128da69be7c9, galera-mariadb-2
=================================================
2024-05-11 0:48:13 2 [Note] WSREP: Non-primary view
2024-05-11 0:48:13 2 [Note] WSREP: Server status change connected -> connected
2024-05-11 0:48:13 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2024-05-11 0:48:18 0 [Warning] WSREP: no nodes coming from prim view, prim not possible
2024-05-11 0:48:18 0 [Note] WSREP: view(view_id(NON_PRIM,9cc9001c-a1ff,19) memb {
9cc9001c-a1ff,0
} joined {
} left {
} partitioned {
})
2024-05-11 0:48:18 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2024-05-11 0:48:18 0 [Note] WSREP: Flow-control interval: [16, 16]
2024-05-11 0:48:18 0 [Note] WSREP: Received NON-PRIMARY.
2024-05-11 0:48:18 2 [Note] WSREP: ================================================
View:
id: 00000000-0000-0000-0000-000000000000:-1
status: non-primary
protocol_version: -1
capabilities:
final: no
own_index: 0
members(1):
0: 9cc9001c-0f27-11ef-a1ff-128da69be7c9, galera-mariadb-2
=================================================
2024-05-11 0:48:18 2 [Note] WSREP: Non-primary view
2024-05-11 0:48:18 2 [Note] WSREP: Server status change connected -> connected
2024-05-11 0:48:18 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2024-05-11 0:48:24 0 [Warning] WSREP: no nodes coming from prim view, prim not possible
2024-05-11 0:48:24 0 [Note] WSREP: view(view_id(NON_PRIM,9cc9001c-a1ff,20) memb {
9cc9001c-a1ff,0
} joined {
} left {
} partitioned {
})
2024-05-11 0:48:24 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2024-05-11 0:48:24 0 [Note] WSREP: Flow-control interval: [16, 16]
2024-05-11 0:48:24 0 [Note] WSREP: Received NON-PRIMARY.
2024-05-11 0:48:24 2 [Note] WSREP: ================================================
View:
id: 00000000-0000-0000-0000-000000000000:-1
status: non-primary
protocol_version: -1
capabilities:
final: no
own_index: 0
members(1):
0: 9cc9001c-0f27-11ef-a1ff-128da69be7c9, galera-mariadb-2
=================================================
2024-05-11 0:48:24 2 [Note] WSREP: Non-primary view
2024-05-11 0:48:24 2 [Note] WSREP: Server status change connected -> connected
2024-05-11 0:48:24 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2024-05-11 0:48:27 0 [Warning] WSREP: Failed to unserialize message. This may be a result of corrupt message, port scanner or another application connecting to group communication port.
2024-05-11 0:48:30 0 [Warning] WSREP: no nodes coming from prim view, prim not possible
2024-05-11 0:48:30 0 [Note] WSREP: view(view_id(NON_PRIM,9cc9001c-a1ff,21) memb {
9cc9001c-a1ff,0
} joined {
} left {
} partitioned {
})
2024-05-11 0:48:30 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2024-05-11 0:48:30 0 [Note] WSREP: Flow-control interval: [16, 16]
2024-05-11 0:48:30 0 [Note] WSREP: Received NON-PRIMARY.
2024-05-11 0:48:30 2 [Note] WSREP: ================================================
View:
id: 00000000-0000-0000-0000-000000000000:-1
status: non-primary
protocol_version: -1
capabilities:
final: no
own_index: 0
members(1):
0: 9cc9001c-0f27-11ef-a1ff-128da69be7c9, galera-mariadb-2
=================================================
2024-05-11 0:48:30 2 [Note] WSREP: Non-primary view
2024-05-11 0:48:30 2 [Note] WSREP: Server status change connected -> connected
2024-05-11 0:48:30 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2024-05-11 0:48:36 0 [Warning] WSREP: no nodes coming from prim view, prim not possible
2024-05-11 0:48:36 0 [Note] WSREP: view(view_id(NON_PRIM,9cc9001c-a1ff,22) memb {
9cc9001c-a1ff,0
} joined {
} left {
} partitioned {
})
2024-05-11 0:48:36 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2024-05-11 0:48:36 0 [Note] WSREP: Flow-control interval: [16, 16]
2024-05-11 0:48:36 0 [Note] WSREP: Received NON-PRIMARY.
2024-05-11 0:48:36 2 [Note] WSREP: ================================================
View:
id: 00000000-0000-0000-0000-000000000000:-1
status: non-primary
protocol_version: -1
capabilities:
final: no
own_index: 0
members(1):
0: 9cc9001c-0f27-11ef-a1ff-128da69be7c9, galera-mariadb-2
=================================================
2024-05-11 0:48:36 2 [Note] WSREP: Non-primary view
2024-05-11 0:48:36 2 [Note] WSREP: Server status change connected -> connected
2024-05-11 0:48:36 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
This is after restoring from S3 with the following section:
bootstrapFrom:
s3:
bucket: mysql-backups
endpoint: minio.minio.svc.newcluster.local:9000
region: us-east-1
accessKeyIdSecretKeyRef:
name: minio-creds
key: MINIO_ACCESS_KEY
secretAccessKeySecretKeyRef:
name: minio-creds
key: MINIO_SECRET_KEY
tls:
enabled: false
restoreJob:
args:
- --verbose
galera:
enabled: true
primary:
automaticFailover: true
replicaThreads: 1
agent:
image: ghcr.io/mariadb-operator/mariadb-operator:v0.0.28
port: 5555
kubernetesAuth:
enabled: true
gracefulShutdownTimeout: 1s
recovery:
enabled: true
minClusterSize: 40%
clusterHealthyTimeout: 30s
clusterBootstrapTimeout: 10m0s
podRecoveryTimeout: 3m0s
podSyncTimeout: 3m0s
initContainer:
image: ghcr.io/mariadb-operator/mariadb-operator:v0.0.28
config:
reuseStorageVolume: false
volumeClaimTemplate:
resources:
requests:
storage: 300Mi
accessModes:
- ReadWriteOnceAll the pods went into a crash loop after 45 minutes after restoring from the above section @mmontes11, this is from an S3 bucket that the operator was writing to before...
perfectra1n commented
Does restoring a Galera cluster work for you? I can't seem to get it to run longer than 30 minutes
perfectra1n commented
and the restore job just runs, and runs, and runs....
mmontes11 commented
All the pods went into a crash loop after 45 minutes after restoring from the above section @mmontes11, this is from an S3 bucket that the operator was writing to before...
If before means before v0.0.28 it is expected, as the backups will contain LOCK TABLES; statements that are incompatible with Galera see:
and the restore job just runs, and runs, and runs....
Once you get rid of the LOCK TABLES;, you should set an adequate number of CPU and memory resources to the Restore Job to make sure it doesn't hang: