Dataset Operator Pod restarting
sakmalh opened this issue · 0 comments
sakmalh commented
Dataset Operator was working fine and then I had to Rollout Restart my kubernetes cluster (I am using Kops). After the restart all my other pods related to dlf works fine. But I get error on this operator pod. I cant assign new pvc and it throws an error. The following error is during it runs.
43364516593096e+09 ERROR controller.dataset Could not wait for Cache to sync {"reconciler group": "com.ie.ibm.hpsys", "reconciler kind": "Dataset", "error": "failed to wait for dataset ca │
│ .k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 │
│ /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:208 │
│ .k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start │
│ /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:234 │
│ .k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1 │
│ /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/manager/runnable_group.go:218 │
│ 43364516593094e+09 ERROR controller.datasetinternal Could not wait for Cache to sync {"reconciler group": "com.ie.ibm.hpsys", "reconciler kind": "DatasetInternal", "error": "failed to wai │
│ .k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 │
│ /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:208 │
│ .k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start │
│ /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:234 │
│ .k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1 │
│ /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/manager/runnable_group.go:218 │
│ 43364516607056e+09 ERROR error received after stop sequence was engaged {"error": "failed to wait for datasetinternal caches to sync: timed out waiting for cache to be synced"} │
│ .k8s.io/controller-runtime/pkg/manager.(*controllerManager).engageStopProcedure.func1 │
│ /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/manager/internal.go:541 │
│ 43364516605525e+09 INFO Stopping and waiting for non leader election runnables │
│ 43364516609583e+09 INFO Stopping and waiting for leader election runnables │
│ 43364516613445e+09 INFO Stopping and waiting for caches │
│ 43364516617835e+09 INFO Stopping and waiting for webhooks │
│ 43364516626759e+09 INFO controller-runtime.webhook shutting down webhook server │
│ 43364516630316e+09 INFO Wait completed, proceeding to shutdown the manager │
│ 43364516631134e+09 ERROR dataset-operator-setup problem running manager {"error": "failed to wait for dataset caches to sync: timed out waiting for cache to be synced"}
The following error is when the pod has failed
│ dataset-operator 1.6943356037081125e+09 ERROR Failed to get API Group-Resources {"error": "Get \"https://*********/api?timeout=32s\": dial tcp *********: connect: no route to host" │
│ dataset-operator sigs.k8s.io/controller-runtime/pkg/cluster.New │
│ dataset-operator /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/cluster/cluster.go:160 │
│ dataset-operator sigs.k8s.io/controller-runtime/pkg/manager.New │
│ dataset-operator /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/manager/manager.go:313 │
│ dataset-operator main.main │
│ dataset-operator /dataset-operator/main.go:76 │
│ dataset-operator runtime.main │
│ dataset-operator /usr/local/go/src/runtime/proc.go:250 │
│ dataset-operator 1.6943356037082324e+09 ERROR dataset-operator-setup unable to start manager {"error": "Get \"https://*********/api?timeout=32s\": dial tcp 1*********: connect: n │
│ dataset-operator main.main │
│ dataset-operator /dataset-operator/main.go:86 │
│ dataset-operator runtime.main │
│ dataset-operator /usr/local/go/src/runtime/proc.go:250