IsNamespaceAnnotated is not thread safe
Closed this issue · 0 comments
eytan-avisror commented
Controller has a watch on namespaces, IsNamespaceAnnotated checks the map of namespaces:annotations whether an annotation exist, however when we read from the map the operation is not thread safe.
In rare cases this may lead to a panic:
2021-04-23T15:57:31.495-0700 INFO namespace watch event {"object": "some-ns"}
fatal error: 2021-04-23T15:57:31.495-0700 INFO namespace watch event {"object": "some-ns-2"}
concurrent map read and map write
fatal error: concurrent map read and map write
goroutine 557 [running]:
runtime.throw(0x2c8be2b, 0x21)
/usr/local/go/src/runtime/panic.go:1117 +0x72 fp=0xc001897330 sp=0xc001897300 pc=0x1038372
runtime.mapaccess2(0x293c840, 0xc0006a5e00, 0xc0018973e0, 0xc000d2d040, 0x99)
/usr/local/go/src/runtime/map.go:469 +0x255 fp=0xc001897370 sp=0xc001897330 pc=0x1010e95
github.com/keikoproj/instance-manager/controllers.(*InstanceGroupReconciler).IsNamespaceAnnotated(0xc0005d9980, 0xc0005a2498, 0x12, 0x2c99c6d, 0x28, 0x2c5d7b0, 0x4, 0x2fe0218)
/Users/eibissror/go/src/github.com/keikoproj/instance-manager/controllers/instancegroup_controller.go:236 +0xc5 fp=0xc001897578 sp=0xc001897370 pc=0x26a3025
We should add locking/unlocking here:
instance-manager/controllers/instancegroup_controller.go
Lines 235 to 236 in 165312e