cainjector pod goes into CrashLoopBackOff status: MutatingWebhookConfiguration
Closed this issue · 3 comments
Hi guys:
my kubernetes version is v1.22.2
i pull this repo, and ready to exec setup.sh
kubectl apply -f ./
deployment.apps/boots created
service/boots created
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io configured
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io configured
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io configured
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io configured
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io configured
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io configured
namespace/cert-manager created
serviceaccount/cert-manager-cainjector created
serviceaccount/cert-manager created
serviceaccount/cert-manager-webhook created
clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrole.rbac.authorization.k8s.io/cert-manager-view created
clusterrole.rbac.authorization.k8s.io/cert-manager-edit created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
role.rbac.authorization.k8s.io/cert-manager:leaderelection created
role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
service/cert-manager created
service/cert-manager-webhook created
deployment.apps/cert-manager-cainjector created
deployment.apps/cert-manager created
deployment.apps/cert-manager-webhook created
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
deployment.apps/db created
service/db created
persistentvolumeclaim/postgres-data created
deployment.apps/dhcrelay created
service/dhcrelay created
configmap/boots created
configmap/dhcrelay created
configmap/tink-client created
configmap/tink-init created
secret/db created
secret/packet created
secret/registry created
secret/tink-auth created
deployment.apps/hegel created
service/hegel created
configmap/my-root-ca.crt created
deployment.apps/nginx created
service/nginx created
persistentvolumeclaim/nginx-data created
deployment.apps/registry created
service/registry created
persistentvolumeclaim/registry-data created
deployment.apps/tink-cli created
job.batch/tink-init created
deployment.apps/tink-server created
service/tink-server created
Error from server (InternalError): error when creating "certs.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.245.222.235:443: connect: connection refused
Error from server (InternalError): error when creating "certs.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.245.222.235:443: connect: connection refused
Error from server (InternalError): error when creating "certs.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.245.222.235:443: connect: connection refused
Error from server (InternalError): error when creating "certs.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.245.222.235:443: connect: connection refused
kubectl get all -n cert-manager
NAME READY STATUS RESTARTS AGE
pod/cert-manager-5597cff495-lgz6f 1/1 Running 0 2d21h
pod/cert-manager-cainjector-bd5f9c764-lb6lj 0/1 CrashLoopBackOff 8 ( ago) 20m
pod/cert-manager-webhook-c4b5687dc-f25ft 1/1 Running 0 2d21h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/cert-manager ClusterIP 10.245.90.139 9402/TCP 2d21h
service/cert-manager-webhook ClusterIP 10.245.188.82 443/TCP 2d21h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/cert-manager 1/1 1 1 2d21h
deployment.apps/cert-manager-cainjector 0/1 1 0 2d21h
deployment.apps/cert-manager-webhook 1/1 1 1 2d21h
NAME DESIRED CURRENT READY AGE
replicaset.apps/cert-manager-5597cff495 1 1 1 2d21h
replicaset.apps/cert-manager-cainjector-bd5f9c764 1 1 0 2d21h
replicaset.apps/cert-manager-webhook-c4b5687dc 1 1 1 2d21h
kubectl logs -f -n cert-manager cert-manager-cainjector-bd5f9c764-lb6lj
I1115 09:02:34.178462 1 start.go:91] "starting" version="v1.1.0" revision="7fbdd6487646e812fe74c0c05503805b5d9d4751"
I1115 09:02:34.737378 1 leaderelection.go:243] attempting to acquire leader lease kube-system/cert-manager-cainjector-leader-election...
I1115 09:02:49.822512 1 leaderelection.go:253] successfully acquired lease kube-system/cert-manager-cainjector-leader-election
I1115 09:02:49.822750 1 recorder.go:52] cert-manager/controller-runtime/manager/events "msg"="Normal" "message"="cert-manager-cainjector-bd5f9c764-lb6lj_e6cd18ac-3ed4-401b-b841-9ff39bef119f became leader" "object"={"kind":"ConfigMap","namespace":"kube-system","name":"cert-manager-cainjector-leader-election","uid":"80007a3f-b9d4-44a1-8ea3-f387d5e47045","apiVersion":"v1","resourceVersion":"6995575"} "reason"="LeaderElection"
E1115 09:02:50.379565 1 start.go:119] cert-manager/ca-injector "msg"="manager goroutine exited" "error"=null
E1115 09:02:50.379611 1 internal.go:521] cert-manager/controller-runtime/manager "msg"="error received after stop sequence was engaged" "error"="leader election lost"
I1115 09:02:51.525023 1 request.go:645] Throttling request took 1.04315391s, request: GET:https://10.245.0.1:443/apis/storage.k8s.io/v1?timeout=32s
E1115 09:02:52.427055 1 start.go:151] cert-manager/ca-injector "msg"="Error registering certificate based controllers. Retrying after 5 seconds." "error"="no matches for kind "MutatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1""
Error: error registering secret controller: no matches for kind "MutatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"
Usage:
ca-injector [flags]
Flags:
--add_dir_header If true, adds the file directory to the header of the log messages
--alsologtostderr log to standard error as well as files
-h, --help help for ca-injector
--kubeconfig string Paths to a kubeconfig. Only required if out-of-cluster.
--leader-elect If true, cainjector will perform leader election between instances to ensure no more than one instance of cainjector operates at a time (default true)
--leader-election-namespace string Namespace used to perform leader election (defaults to controller's namespace). Only used if leader election is enabled
--log-flush-frequency duration Maximum number of seconds between log flushes (default 5s)
--log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
--log_dir string If non-empty, write log files in this directory
--log_file string If non-empty, use this log file
--log_file_max_size uint Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
--logtostderr log to standard error instead of files (default true)
--master --kubeconfig (Deprecated: switch to --kubeconfig) The address of the Kubernetes API server. Overrides any value in kubeconfig. Only required if out-of-cluster.
--namespace string If set, this limits the scope of cainjector to a single namespace. If set, cainjector will not update resources with certificates outside of the configured namespace.
--skip_headers If true, avoid header prefixes in the log messages
--skip_log_headers If true, avoid headers when opening log files
--stderrthreshold severity logs at or above this threshold go to stderr (default 2)
-v, --v Level number for the log level verbosity (default 0)
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
error registering secret controller: no matches for kind "MutatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"
[root@k8s-node2 ~]# docker logs -f 20f6831a09c0
I1116 10:29:16.890958 1 start.go:91] "starting" version="v1.1.0" revision="7fbdd6487646e812fe74c0c05503805b5d9d4751"
I1116 10:29:17.346901 1 leaderelection.go:243] attempting to acquire leader lease kube-system/cert-manager-cainjector-leader-election...
I1116 10:29:33.351547 1 leaderelection.go:253] successfully acquired lease kube-system/cert-manager-cainjector-leader-election
I1116 10:29:33.352379 1 recorder.go:52] cert-manager/controller-runtime/manager/events "msg"="Normal" "message"="cert-manager-cainjector-bd5f9c764-qgtvx_8d896baa-ec4d-481a-85d5-466b09e6c66c became leader" "object"={"kind":"ConfigMap","namespace":"kube-system","name":"cert-manager-cainjector-leader-election","uid":"55682406-8a67-4577-ad1c-23c5043c543f","apiVersion":"v1","resourceVersion":"11846"} "reason"="LeaderElection"
E1116 10:29:33.804117 1 start.go:119] cert-manager/ca-injector "msg"="manager goroutine exited" "error"=null
I1116 10:29:34.952203 1 request.go:645] Throttling request took 1.048271738s, request: GET:https://10.96.0.1:443/apis/admissionregistration.k8s.io/v1?timeout=32s
E1116 10:29:35.754094 1 start.go:151] cert-manager/ca-injector "msg"="Error registering certificate based controllers. Retrying after 5 seconds." "error"="no matches for kind "MutatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1""
Error: error registering secret controller: no matches for kind "MutatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"
Usage:
ca-injector [flags]
Flags:
--add_dir_header If true, adds the file directory to the header of the log messages
--alsologtostderr log to standard error as well as files
-h, --help help for ca-injector
--kubeconfig string Paths to a kubeconfig. Only required if out-of-cluster.
--leader-elect If true, cainjector will perform leader election between instances to ensure no more than one instance of cainjector operates at a time (default true)
--leader-election-namespace string Namespace used to perform leader election (defaults to controller's namespace). Only used if leader election is enabled
--log-flush-frequency duration Maximum number of seconds between log flushes (default 5s)
--log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
--log_dir string If non-empty, write log files in this directory
--log_file string If non-empty, use this log file
--log_file_max_size uint Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
--logtostderr log to standard error instead of files (default true)
--master --kubeconfig (Deprecated: switch to --kubeconfig) The address of the Kubernetes API server. Overrides any value in kubeconfig. Only required if out-of-cluster.
--namespace string If set, this limits the scope of cainjector to a single namespace. If set, cainjector will not update resources with certificates outside of the configured namespace.
--skip_headers If true, avoid header prefixes in the log messages
--skip_log_headers If true, avoid headers when opening log files
--stderrthreshold severity logs at or above this threshold go to stderr (default 2)
-v, --v Level number for the log level verbosity (default 0)
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
error registering secret controller: no matches for kind "MutatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"
any idea? with great appreciate.
https://kubernetes.io/blog/2021/07/14/upcoming-changes-in-kubernetes-1-22/
It seems that this API was removed:
Beta versions of the ValidatingWebhookConfiguration and MutatingWebhookConfiguration API (the admissionregistration.k8s.io/v1beta1 API versions)
I don't have a Kubernetes environment handy for replication - but can you try replacing deploy/kubernetes/cert-manager.yaml with https://github.com/jetstack/cert-manager/releases/download/v1.6.1/cert-manager.yaml
I'm relatively confident that it will help to upgrade to the latest cert-manager.
https://kubernetes.io/blog/2021/07/14/upcoming-changes-in-kubernetes-1-22/
It seems that this API was removed:
Beta versions of the ValidatingWebhookConfiguration and MutatingWebhookConfiguration API (the admissionregistration.k8s.io/v1beta1 API versions)
I don't have a Kubernetes environment handy for replication - but can you try replacing
deploy/kubernetes/cert-manager.yamlwith https://github.com/jetstack/cert-manager/releases/download/v1.6.1/cert-manager.yamlI'm relatively confident that it will help to upgrade to the latest cert-manager.
Thank you very much @tstromberg
i used the new version of cert-manager and the issue is resolved.
my k8s version is: v1.22.3
kk api-resources |grep admissionregistration
mutatingwebhookconfigurations admissionregistration.k8s.io/v1 false MutatingWebhookConfiguration
validatingwebhookconfigurations admissionregistration.k8s.io/v1 false ValidatingWebhookConfiguration
Re-opening until we can solve the underlying problem.