Internal error occurred: failed calling webhook "mcomposable.kb.io"
Closed this issue · 2 comments
lhriley commented
I have installed everything, including the webhooks and cert-manager resources. However, when I try to apply a manifest with composable resources, I get the following error:
Error:
failed to create resource: Internal error occurred: failed calling webhook "mcomposable.kb.io": failed to call webhook: Post "https://composable-webhook-service.composable-system.svc:443/mutate-ibmcloud-ibm-com-v1alpha1-composable?timeout=10s": context deadline exceeded
Install logs:
Installing composable-operator
namespace/composable-system unchanged
customresourcedefinition.apiextensions.k8s.io/composables.ibmcloud.ibm.com configured
serviceaccount/composable-controller-manager unchanged
role.rbac.authorization.k8s.io/composable-leader-election-role unchanged
clusterrole.rbac.authorization.k8s.io/composable-manager-role configured
clusterrole.rbac.authorization.k8s.io/composable-metrics-reader unchanged
clusterrole.rbac.authorization.k8s.io/composable-proxy-role unchanged
rolebinding.rbac.authorization.k8s.io/composable-leader-election-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/composable-manager-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/composable-proxy-rolebinding unchanged
service/composable-controller-manager-metrics-service unchanged
service/composable-webhook-service unchanged
deployment.apps/composable-controller-manager unchanged
certificate.cert-manager.io/composable-serving-cert unchanged
issuer.cert-manager.io/composable-selfsigned-issuer unchanged
mutatingwebhookconfiguration.admissionregistration.k8s.io/composable-mutating-webhook-configuration configured
validatingwebhookconfiguration.admissionregistration.k8s.io/composable-validating-webhook-configuration configure
Resources:
❯ kubectl get all -n composable-system
NAME READY STATUS RESTARTS AGE
pod/composable-controller-manager-5cdf4fcb-8xjjk 2/2 Running 1 (2d18h ago) 2d20h
pod/composable-controller-manager-6bb958c567-n7htx 1/2 ImagePullBackOff 0 2d18h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/composable-controller-manager-metrics-service ClusterIP 10.7.217.238 <none> 8443/TCP 2d20h
service/composable-webhook-service ClusterIP 10.7.16.143 <none> 443/TCP 2d20h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/composable-controller-manager 1/1 1 1 2d20h
NAME DESIRED CURRENT READY AGE
replicaset.apps/composable-controller-manager-5cdf4fcb 1 1 1 2d20h
replicaset.apps/composable-controller-manager-6bb958c567 1 1 0 2d18h
❯ kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io | grep composable
composable-mutating-webhook-configuration 1 2d20h
❯ kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io | grep composable
composable-validating-webhook-configuration 1 2d20h
❯ kubectl get certificates.cert-manager.io -n composable-system
NAME READY SECRET AGE
composable-serving-cert True webhook-server-cert 2d20h
The new pod that fails to start has this status:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 38s default-scheduler Successfully assigned composable-system/composable-controller-manager-6bb958c567-44hct to gke-env0-default-0e0b52e6-xdfr
Normal Pulled 31s kubelet Container image "gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0" already present on machine
Normal Created 31s kubelet Created container kube-rbac-proxy
Normal Started 31s kubelet Started container kube-rbac-proxy
Normal Pulling 15s (x2 over 37s) kubelet Pulling image "controller:latest"
Warning Failed 14s (x2 over 31s) kubelet Failed to pull image "controller:latest": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/controller:latest": failed to resolve reference "docker.io/library/controller:latest": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Warning Failed 14s (x2 over 31s) kubelet Error: ErrImagePull
Normal BackOff 1s (x3 over 31s) kubelet Back-off pulling image "controller:latest"
Warning Failed 1s (x3 over 31s) kubelet Error: ImagePullBackOff
This image appears to not exist:
❯ docker pull docker.io/library/controller:latest
Error response from daemon: pull access denied for controller, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
Upon further investigation I found this is the source of the bad container image ref:
--- a/config/manager/manager.yaml
+++ b/config/manager/manager.yaml
@@ -31,7 +31,7 @@ spec:
- /manager
args:
- --leader-elect
- image: controller:latest
+ image: ghcr.io/composable-operator/composable:0.2.1
name: manager
securityContext:
allowPrivilegeEscalation: false
lhriley commented
Figured it out! This is due to firewall rules in GCP / GKE. We had webhook port 8443 allowed, but not port 9443.
Reference: open-telemetry/opentelemetry-operator#1009