emqx-operator-controller-manager pod failed with runtime error: index out of range

Question

emqx-operator-controller-manager pod failed with runtime error: index out of range

suojun-hao opened this issue 2 years ago · 2 comments

Describe the bug
emqx-operator-controller-manager pod keep crashing with the below logs after deploy the emqx opensource broker to cluster. here we use kustomize.

Logs:

I1103 14:44:33.965126       1 request.go:601] Waited for 1.003207335s due to client-side throttling, not priority and fairness, request: GET:https://172.0.0.1:443/apis/scheduling.k8s.io/v1?timeout=32s
65
2022-11-03T14:44:35Z	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": ":8080"}
64
2022-11-03T14:44:35Z	INFO	controller-runtime.builder	Registering a mutating webhook	{"GVK": "apps.emqx.io/v1beta3, Kind=EmqxPlugin", "path": "/mutate-apps-emqx-io-v1beta3-emqxplugin"}
63
2022-11-03T14:44:35Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/mutate-apps-emqx-io-v1beta3-emqxplugin"}
62
2022-11-03T14:44:35Z	INFO	controller-runtime.builder	Registering a validating webhook	{"GVK": "apps.emqx.io/v1beta3, Kind=EmqxPlugin", "path": "/validate-apps-emqx-io-v1beta3-emqxplugin"}
61
2022-11-03T14:44:35Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/validate-apps-emqx-io-v1beta3-emqxplugin"}
60
2022-11-03T14:44:35Z	INFO	controller-runtime.builder	Registering a mutating webhook	{"GVK": "apps.emqx.io/v1beta3, Kind=EmqxBroker", "path": "/mutate-apps-emqx-io-v1beta3-emqxbroker"}
59
2022-11-03T14:44:35Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/mutate-apps-emqx-io-v1beta3-emqxbroker"}
58
2022-11-03T14:44:35Z	INFO	controller-runtime.builder	Registering a validating webhook	{"GVK": "apps.emqx.io/v1beta3, Kind=EmqxBroker", "path": "/validate-apps-emqx-io-v1beta3-emqxbroker"}
57
2022-11-03T14:44:35Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/validate-apps-emqx-io-v1beta3-emqxbroker"}
56
2022-11-03T14:44:35Z	INFO	controller-runtime.builder	Registering a mutating webhook	{"GVK": "apps.emqx.io/v1beta3, Kind=EmqxEnterprise", "path": "/mutate-apps-emqx-io-v1beta3-emqxenterprise"}
55
2022-11-03T14:44:35Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/mutate-apps-emqx-io-v1beta3-emqxenterprise"}
54
2022-11-03T14:44:35Z	INFO	controller-runtime.builder	Registering a validating webhook	{"GVK": "apps.emqx.io/v1beta3, Kind=EmqxEnterprise", "path": "/validate-apps-emqx-io-v1beta3-emqxenterprise"}
53
2022-11-03T14:44:35Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/validate-apps-emqx-io-v1beta3-emqxenterprise"}
52
2022-11-03T14:44:35Z	INFO	controller-runtime.builder	Registering a mutating webhook	{"GVK": "apps.emqx.io/v2alpha1, Kind=EMQX", "path": "/mutate-apps-emqx-io-v2alpha1-emqx"}
51
2022-11-03T14:44:35Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/mutate-apps-emqx-io-v2alpha1-emqx"}
50
2022-11-03T14:44:35Z	INFO	controller-runtime.builder	Registering a validating webhook	{"GVK": "apps.emqx.io/v2alpha1, Kind=EMQX", "path": "/validate-apps-emqx-io-v2alpha1-emqx"}
49
2022-11-03T14:44:35Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/validate-apps-emqx-io-v2alpha1-emqx"}
48
2022-11-03T14:44:35Z	INFO	setup	starting manager
47
2022-11-03T14:44:35Z	INFO	controller-runtime.webhook.webhooks	Starting webhook server
46
2022-11-03T14:44:35Z	INFO	Starting server	{"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
45
2022-11-03T14:44:35Z	INFO	Starting server	{"kind": "health probe", "addr": "[::]:8081"}
44
2022-11-03T14:44:35Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
43
I1103 14:44:35.824603       1 leaderelection.go:248] attempting to acquire leader lease emqx/19fd6fcc.emqx.io...
42
2022-11-03T14:44:35Z	INFO	controller-runtime.certwatcher	Starting certificate watcher
41
2022-11-03T14:44:35Z	INFO	controller-runtime.webhook	Serving webhook server	{"host": "", "port": 9443}
40
I1103 14:45:07.039257       1 leaderelection.go:258] successfully acquired lease emqx/19fd6fcc.emqx.io
39
2022-11-03T14:45:07Z	INFO	Starting EventSource	{"controller": "emqxplugin", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxPlugin", "source": "kind source: *v1beta3.EmqxPlugin"}
38
2022-11-03T14:45:07Z	INFO	Starting EventSource	{"controller": "emqxbroker", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxBroker", "source": "kind source: *v1beta3.EmqxBroker"}
37
2022-11-03T14:45:07Z	INFO	Starting Controller	{"controller": "emqxbroker", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxBroker"}
36
2022-11-03T14:45:07Z	INFO	Starting EventSource	{"controller": "emqx", "controllerGroup": "apps.emqx.io", "controllerKind": "EMQX", "source": "kind source: *v2alpha1.EMQX"}
35
2022-11-03T14:45:07Z	INFO	Starting Controller	{"controller": "emqx", "controllerGroup": "apps.emqx.io", "controllerKind": "EMQX"}
34
2022-11-03T14:45:07Z	INFO	Starting Controller	{"controller": "emqxplugin", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxPlugin"}
33
2022-11-03T14:45:07Z	INFO	Starting EventSource	{"controller": "emqxenterprise", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxEnterprise", "source": "kind source: *v1beta3.EmqxEnterprise"}
32
2022-11-03T14:45:07Z	INFO	Starting Controller	{"controller": "emqxenterprise", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxEnterprise"}
31
2022-11-03T14:45:07Z	INFO	Starting workers	{"controller": "emqxplugin", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxPlugin", "worker count": 1}
30
2022-11-03T14:45:07Z	INFO	Starting workers	{"controller": "emqxbroker", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxBroker", "worker count": 1}
29
2022-11-03T14:45:07Z	INFO	Starting workers	{"controller": "emqx", "controllerGroup": "apps.emqx.io", "controllerKind": "EMQX", "worker count": 1}
28
2022-11-03T14:45:07Z	INFO	Starting workers	{"controller": "emqxenterprise", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxEnterprise", "worker count": 1}
27
2022-11-03T14:45:07Z	INFO	Observed a panic in reconciler: runtime error: index out of range [0] with length 0	{"controller": "emqx", "controllerGroup": "apps.emqx.io", "controllerKind": "EMQX", "eMQX": {"name":"mycluster1-emqx","namespace":"emqx"}, "namespace": "emqx", "name": "mycluster1-emqx", "reconcileID": "7b73e597-435a-45b0-b9e5-57fe41c34757"}
26
panic: runtime error: index out of range [0] with length 0 [recovered]
25
	panic: runtime error: index out of range [0] with length 0
24
23
goroutine 492 [running]:
22
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
21
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:118 +0x1f4
20
panic({0x1847520, 0xc001c7a030})
19
	/usr/local/go/src/runtime/panic.go:838 +0x207
18
github.com/emqx/emqx-operator/controllers/apps/v2alpha1.(*coreUpdateStatus).nextStatus(0xc0003240f0, 0xc00015ca00, 0xc00011a800?)
17
	/workspace/controllers/apps/v2alpha1/status_machine.go:163 +0x2c5
16
github.com/emqx/emqx-operator/controllers/apps/v2alpha1.(*emqxStatusMachine).NextStatus(...)
15
	/workspace/controllers/apps/v2alpha1/status_machine.go:111
14
github.com/emqx/emqx-operator/controllers/apps/v2alpha1.(*EMQXReconciler).updateStatus(0xc000601c00, 0xc00011a800)
13
	/workspace/controllers/apps/v2alpha1/emqx_controller.go:160 +0x2f0
12
github.com/emqx/emqx-operator/controllers/apps/v2alpha1.(*EMQXReconciler).Reconcile(0xc000601c00, {0x1ba82c8, 0xc00086d020}, {{{0xc000404630?, 0x10?}, {0xc000404620?, 0x40d7e7?}}})
11
	/workspace/controllers/apps/v2alpha1/emqx_controller.go:75 +0x134
10
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1ba8220?, {0x1ba82c8?, 0xc00086d020?}, {{{0xc000404630?, 0x1878040?}, {0xc000404620?, 0x4041f4?}}})
9
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121 +0xc8
8
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0001f1680, {0x1ba8220, 0xc000932580}, {0x179bae0?, 0xc000529380?})
7
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320 +0x33c
6
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0001f1680, {0x1ba8220, 0xc000932580})
5
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273 +0x1d9
4
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
3
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234 +0x85
2
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
1
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:230 +0x325

Environment details::

Kubernetes version: 1.24.6
Cloud-provider/provisioner: Azure
emqx-operator version: 2.0.1
Install method: static manifests

Answer 1 · 2022-11-04T01:33:30.000Z

Sorry about, we will fix this in next release

Answer 2 · 2022-11-04T09:26:28.000Z

Sorry about, we will fix this in next release

okay, thanks a lot. we will try with the new release.