Koperator panics when CruiseControl responds with a progress/async response
amuraru opened this issue · 0 comments
amuraru commented
Describe the bug
CruiseControl has webserver.request.maxBlockTimeMs
configuration that controls the threshold above which a request is converted to async and a handler returned.
koperator is not able to handle this and panics:
{"level":"info","ts":"2022-06-29T05:24:46.382Z","logger":"controller.CruiseControl","msg":"got response for request","reconciler group":"kafka.banzaicloud.io","reconciler kind":"KafkaCluster","name":"pipeline-kafka","namespace":"ns-team-aep-pipeline-kafka-1-int","url":"http://pipeline-kafka-cruisecontrol-svc.ns-team-aep-pipeline-kafka-1-int.svc.cluster.local:8090/kafkacruisecontrol/state?json=true&substates=ANALYZER%2CANOMALY_DETECTOR%2CEXECUTOR%2CMONITOR&verbose=true","status":202}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x2e8 pc=0x16dc126]
goroutine 1368 [running]:
github.com/banzaicloud/koperator/pkg/scale.(*cruiseControlScaler).Status(0xc002a49bf0)
/workspace/pkg/scale/scale.go:75 +0x186
github.com/banzaicloud/koperator/controllers.(*CruiseControlTaskReconciler).Reconcile(0xc000304b58, {0x2143e20, 0xc002a48c60}, {{{0xc00068f940?, 0x1d65200?}, {0xc000973ac0?, 0x30?}}})
/workspace/controllers/cruisecontroltask_controller.go:91 +0x28a
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0xc00056a000, {0x2143e20, 0xc002a48ba0}, {{{0xc00068f940?, 0x1d65200?}, {0xc000973ac0?, 0x4041f4?}}})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114 +0x27e
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00056a000, {0x2143d78, 0xc000b91800}, {0x1c15100?, 0xc000ab6040?})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311 +0x349
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00056a000, {0x2143d78, 0xc000b91800})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266 +0x1d9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:223 +0x31c
Steps to reproduce the issue:
Hard to reproduce but happens when CruiseControl is loaded (slow)
Expected behavior
koperator should gracefully handle this case.
Screenshots
N/A