Nil pointer exception with clusterLogging.Spec.Collection empty and Forwarder enabled
Closed this issue · 3 comments
andreaskaris commented
Nil pointer exception with clusterLogging.Spec.Collection empty and Forwarder enabled.
The following combination throws a nil pointer exception:
ClusterLogging:
apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
metadata:
name: "instance"
namespace: openshift-logging
spec:
managementState: "Managed"
ClusterLogForwarder:
apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
name: instance
namespace: openshift-logging
spec:
pipelines:
- inputRefs:
- audit
outputRefs:
- loki-external
labels:
job: openshift-audit
clusterid: sno10
- inputRefs:
- application
outputRefs:
- loki-external
labels:
job: openshift-application
clusterid: sno10
- inputRefs:
- infrastructure
outputRefs:
- loki-external
labels:
job: openshift-infrastructure
clusterid: sno10
outputs:
- name: loki-external
type: loki
url: http://192.168.18.75:310
[akaris@workstation logging]$ oc apply -f configuring-cluster-logging-minimal-minimal.yaml
clusterlogging.logging.openshift.io/instance created
[akaris@workstation logging]$ oc apply -f configuring-log-forwarder.yaml
clusterlogforwarder.logging.openshift.io/instance created
[akaris@workstation logging]$
$ LOG_LEVEL=3 make run
(...)
{"_ts":"2024-01-18T11:18:45.623084752Z","_file:line":"internal/reconcile/rbac.go:37","_level":"3","_component":"cluster-logging-operator","_message":"Roles are the same skipping update"}
{"_ts":"2024-01-18T11:18:45.623126639Z","_file:line":"internal/reconcile/rbac.go:70","_level":"3","_component":"cluster-logging-operator","_message":"RoleBindings are the same skipping update"}
{"_ts":"2024-01-18T11:18:45.623159759Z","_file:line":"internal/k8shandler/clusterloggingrequest.go:107","_level":"3","_component":"cluster-logging-operator","_message":"Getting object","namespacedName":{"Namespace":"openshift-logging","Name":"logcollector-token"},"object":{"metadata":{"creationTimestamp":null}}}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x15c31d8]
goroutine 351 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
/home/akaris/development/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:119 +0x1e5
panic({0x1737be0?, 0x28a4ed0?})
/usr/lib/golang/src/runtime/panic.go:914 +0x21f
github.com/openshift/cluster-logging-operator/internal/k8shandler.(*ClusterLoggingRequest).generateCollectorConfig(0xc000927058)
/home/akaris/development/cluster-logging-operator/internal/k8shandler/forwarding.go:48 +0x298
github.com/openshift/cluster-logging-operator/internal/k8shandler.(*ClusterLoggingRequest).CreateOrUpdateCollection(0xc000927058)
/home/akaris/development/cluster-logging-operator/internal/k8shandler/collection.go:72 +0x2b2
github.com/openshift/cluster-logging-operator/internal/k8shandler.Reconcile(0xc0008d84e0, 0xc000898a80, {0x1c12238, 0xc000290180}, {0x7fb45823e568, 0xc000292770}, {0x1c099b0, 0xc0002d8340}, {0xc0003ca158, 0x6}, ...)
/home/akaris/development/cluster-logging-operator/internal/k8shandler/reconciler.go:67 +0x81f
github.com/openshift/cluster-logging-operator/controllers/forwarding.(*ReconcileForwarder).Reconcile(0xc000700420, {0x30?, 0xc000588000?}, {{{0xc00063dfb0?, 0x0?}, {0xc000331060?, 0x412205?}}})
/home/akaris/development/cluster-logging-operator/controllers/forwarding/forwarding_controller.go:115 +0xc35
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1c0ab18?, {0x1c0ab18?, 0xc000b04d50?}, {{{0xc00063dfb0?, 0x1684940?}, {0xc000331060?, 0xc000c037d0?}}})
/home/akaris/development/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:122 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000298460, {0x1c0ab50, 0xc0002a90e0}, {0x17b7fe0?, 0xc000332da0?})
/home/akaris/development/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:323 +0x353
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000298460, {0x1c0ab50, 0xc0002a90e0})
/home/akaris/development/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274 +0x1c9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
/home/akaris/development/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 107
/home/akaris/development/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:231 +0x565
exit status 2
make: *** [Makefile:150: run] Error 1
andreaskaris commented
/assign
jasaltvik commented
Hi @andreaskaris. We just upgraded to OpenShift Logging 5.8 (from 5.7), and are now experiencing this. We are using our own custom Vector, so the collection config stanza is removed. We had an unused ClusterLogForwarder in the cluster, but this is now removed. The cluster logging operator still fails with the same error. Is this expected? If yes, are there any workarounds for this (before #2313 is merged and released)?
Current error:
{"_ts":"2024-02-07T11:24:08.571053532Z","_level":"0","_component":"cluster-logging-operator","_message":"starting up...","go_arch":"amd64","go_os":"linux","go_version":"go1.20.10","operator_version":"5.8.0"}
I0207 11:24:09.623336 1 request.go:690] Waited for 1.028887056s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/broker.amq.io/v2alpha3?timeout=32s
{"_ts":"2024-02-07T11:24:14.686185523Z","_level":"0","_component":"cluster-logging-operator","_message":"migrating resources provided by the manifest"}
{"_ts":"2024-02-07T11:24:14.689878193Z","_level":"0","_component":"cluster-logging-operator","_message":"Registering Components."}
{"_ts":"2024-02-07T11:24:14.705099482Z","_level":"0","_component":"cluster-logging-operator","_message":"Starting the Cmd."}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x16b3a38]
goroutine 241 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
/remote-source/cluster-logging-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:119 +0x1fa
panic({0x183a2a0, 0x2946b30})
/usr/lib/golang/src/runtime/panic.go:884 +0x213
github.com/openshift/cluster-logging-operator/internal/k8shandler.(*ClusterLoggingRequest).generateCollectorConfig(0xc000a4cdb8)
/remote-source/cluster-logging-operator/app/internal/k8shandler/forwarding.go:50 +0x298
github.com/openshift/cluster-logging-operator/internal/k8shandler.(*ClusterLoggingRequest).CreateOrUpdateCollection(0xc000a4cdb8)
/remote-source/cluster-logging-operator/app/internal/k8shandler/collection.go:72 +0x2b2
github.com/openshift/cluster-logging-operator/internal/k8shandler.Reconcile(0xc000698340, 0xc0000f8fc0, {0x1d16c38, 0xc00064c180}, {0x7f17183a6e28, 0xc0000f6230}, {0x1d0e5b0, 0xc0005e2040}, {0xc0005fc1e9, 0x7}, ...)
/remote-source/cluster-logging-operator/app/internal/k8shandler/reconciler.go:67 +0x845
github.com/openshift/cluster-logging-operator/controllers/clusterlogging.(*ReconcileClusterLogging).Reconcile(0xc0005f62a0, {0x1d0f0e0, 0xc0008904b0}, {{{0xc0003bd050?, 0x11?}, {0xc00061ee58?, 0x8?}}})
/remote-source/cluster-logging-operator/app/controllers/clusterlogging/clusterlogging_controller.go:120 +0xc87
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1d0f0e0?, {0x1d0f0e0?, 0xc0008904b0?}, {{{0xc0003bd050?, 0x17bd100?}, {0xc00061ee58?, 0x0?}}})
/remote-source/cluster-logging-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:122 +0xc8
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0003f41e0, {0x1d0f038, 0xc0002d6ff0}, {0x18c0e60?, 0xc000456b00?})
/remote-source/cluster-logging-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:323 +0x35f
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0003f41e0, {0x1d0f038, 0xc0002d6ff0})
/remote-source/cluster-logging-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274 +0x1d9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
/remote-source/cluster-logging-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
/remote-source/cluster-logging-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:231 +0x587
jasaltvik commented
@andreaskaris Hmm, it actually seems to work when adding an empty collection config stanza
apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
name: instance
namespace: openshift-logging
spec:
collection: {}
So it seems all is good now.