keikoproj/active-monitor

Active Monitor crashing with concurrent map updates

RaviHari opened this issue · 0 comments

Describe the bug
Active-Monitor is crashing intermittently with this error:

fatal error: concurrent map writes
goroutine 105843 [running]:
runtime.throw(0x1633f60, 0x15)
/usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc00056baa8 sp=0xc00056ba78 pc=0x436532
runtime.mapassign_faststr(0x1488fa0, 0xc00048a1b0, 0xc0006e2ee0, 0x1b, 0xc0008362a0)
/usr/local/go/src/runtime/map_faststr.go:291 +0x3d8 fp=0xc00056bb10 sp=0xc00056baa8 pc=0x414538
github.com/keikoproj/active-monitor/controllers.(*HealthCheckReconciler).watchWorkflowReschedule(0xc0002530e0, 0x17e9da0, 0xc000126200, 0x0, 0x0, 0x0, 0x0, 0x17f1d20, 0xc0008362a0, 0xc000681d80, ...)
/workspace/controllers/healthcheck_controller.go:676 +0x12cf fp=0xc00056bf00 sp=0xc00056bb10 pc=0x132f2ef
github.com/keikoproj/active-monitor/controllers.(*HealthCheckReconciler).createSubmitWorkflowHelper.func1()
/workspace/controllers/healthcheck_controller.go:459 +0x1b5 fp=0xc00056bfe0 sp=0xc00056bf00 pc=0x13394d5
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc00056bfe8 sp=0xc00056bfe0 pc=0x46b941
created by time.goFunc
/usr/local/go/src/time/sleep.go:167 +0x45
goroutine 1 [select, 1658 minutes]:
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).Start(0xc000330a80, 0xc0000fa9c0, 0x0, 0x0)
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/manager/internal.go:433 +0x1dd
main.main()
/workspace/main.go:84 +0x59c