hvpa-controller is in CrashLoopBackOff
ialidzhikov opened this issue · 1 comments
ialidzhikov commented
Describe the bug:
I am observing the following panic of hvpa-controller on our dev env.
$ k -n garden get po hvpa-controller-78c499f8c4-hqv8d
NAME READY STATUS RESTARTS AGE
hvpa-controller-78c499f8c4-hqv8d 0/1 CrashLoopBackOff 17 99m
I1120 14:02:18.315365 1 hvpa_controller.go:1329] controller/hvpa "level"=1 "msg"="Reconciling" "hvpa"="shoot--it--tmz8f-28e/etcd-main"
E1120 14:02:18.341472 1 runtime.go:69] Observed a panic: &runtime.TypeAssertionError{_interface:(*runtime._type)(0x12d4540), concrete:(*runtime._type)(0x13da140), asserted:(*runtime._type)(0x1417c80), missingMethod:""} (interface conversion: runtime.Object is *unstructured.UnstructuredList, not *unstructured.Unstructured)
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/runtime/runtime.go:76
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/runtime/runtime.go:65
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/panic.go:522
/usr/local/go/src/runtime/iface.go:248
/usr/local/go/src/runtime/iface.go:258
/go/pkg/mod/k8s.io/client-go@v11.0.1-0.20190409021438-1a26190bd76a+incompatible/dynamic/simple.go:231
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0-beta.5/pkg/client/unstructured_client.go:147
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0-beta.5/pkg/client/client.go:168
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0-beta.5/pkg/client/split.go:49
/workspace/controllers/hvpa_controller.go:158
/workspace/controllers/hvpa_controller.go:215
/workspace/controllers/hvpa_controller.go:1337
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0-beta.5/pkg/internal/controller/controller.go:216
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0-beta.5/pkg/internal/controller/controller.go:192
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0-beta.5/pkg/internal/controller/controller.go:171
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88
/usr/local/go/src/runtime/asm_amd64.s:1337
panic: interface conversion: runtime.Object is *unstructured.UnstructuredList, not *unstructured.Unstructured [recovered]
panic: interface conversion: runtime.Object is *unstructured.UnstructuredList, not *unstructured.Unstructured
goroutine 421 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/runtime/runtime.go:58 +0x105
panic(0x12c47a0, 0xc002e544b0)
/usr/local/go/src/runtime/panic.go:522 +0x1b5
k8s.io/client-go/dynamic.(*dynamicResourceClient).Get(0xc0048b2320, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/pkg/mod/k8s.io/client-go@v11.0.1-0.20190409021438-1a26190bd76a+incompatible/dynamic/simple.go:231 +0x919
sigs.k8s.io/controller-runtime/pkg/client.(*unstructuredClient).Get(0xc0002b6588, 0x16208a0, 0xc000046240, 0xc0002a9b00, 0x14, 0x0, 0x0, 0x1601c60, 0xc000e8a0c8, 0x1295e20, ...)
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0-beta.5/pkg/client/unstructured_client.go:147 +0x23a
sigs.k8s.io/controller-runtime/pkg/client.(*client).Get(0xc0002b64e0, 0x16208a0, 0xc000046240, 0xc0002a9b00, 0x14, 0x0, 0x0, 0x1601c60, 0xc000e8a0c8, 0x0, ...)
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0-beta.5/pkg/client/client.go:168 +0xaa
sigs.k8s.io/controller-runtime/pkg/client.(*DelegatingReader).Get(0xc000544700, 0x16208a0, 0xc000046240, 0xc0002a9b00, 0x14, 0x0, 0x0, 0x1601c60, 0xc000e8a0c8, 0x0, ...)
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0-beta.5/pkg/client/split.go:49 +0xaa
github.com/gardener/hvpa-controller/controllers.(*HvpaReconciler).getSelectorFromHvpa(0xc0004b1fb0, 0xc0045b9080, 0x2145228, 0x0, 0x0, 0x0)
/workspace/controllers/hvpa_controller.go:158 +0x1fb
github.com/gardener/hvpa-controller/controllers.(*HvpaReconciler).ManageCache(0xc0004b1fb0, 0xc0045b9080, 0xc0002a9b00, 0x14, 0xc0004d3e40, 0x9, 0x1)
/workspace/controllers/hvpa_controller.go:215 +0x99
github.com/gardener/hvpa-controller/controllers.(*HvpaReconciler).Reconcile(0xc0004b1fb0, 0xc0002a9b00, 0x14, 0xc0004d3e40, 0x9, 0x2127960, 0x42bd21, 0x1603bc0, 0xc003f19d88)
/workspace/controllers/hvpa_controller.go:1337 +0x66f
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0000e28c0, 0x12ff160, 0xc000ba0680, 0x12ff100)
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0-beta.5/pkg/internal/controller/controller.go:216 +0x149
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0000e28c0, 0xc004382600)
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0-beta.5/pkg/internal/controller/controller.go:192 +0xb5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc0000e28c0)
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0-beta.5/pkg/internal/controller/controller.go:171 +0x2b
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0021528e0)
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152 +0x54
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0021528e0, 0x3b9aca00, 0x0, 0x1, 0xc000674a20)
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(0xc0021528e0, 0x3b9aca00, 0xc000674a20)
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0-beta.5/pkg/internal/controller/controller.go:157 +0x311
Expected behaviour:
No panic.
How To Reproduce (as minimally and precisely as possible):
Not clear for now.
Anything else we need to know?:
hvpa-controller version: eu.gcr.io/gardener-project/gardener/hvpa-controller:v0.3.1
amshuman-kr commented
The offending line in hvpa_controller.go looks right to me. Get call should be passed an Unstructured and not an UnstructuredList. https://github.com/gardener/hvpa-controller/blob/master/controllers/hvpa_controller.go#L158.
Has there been any change on the apiserver side?