ErrImagePull for Noobaa CLI v.5.12.4
Opened this issue ยท 4 comments
Environment info
- NooBaa Operator Version: v5.12.4
- Platform: Kubernetes, MicroK8s v1.26.7
Actual behavior
I'm doing a green field installation of the Noobaa Operator via the Noobaa CLI, and running into ErrImagePull. I installed Noobaa CLI v5.12.4 but the deployment is trying to pull container image v5.12.0, which does not exist on Docker Hub.
Expected behavior
It should pull v5.12.4
Steps to reproduce
[jonathan@latitude ~]$ brew install noobaa/noobaa/noobaa
Running `brew update --auto-update`...
==> Homebrew collects anonymous analytics.
Read the analytics documentation (and how to opt-out) here:
https://docs.brew.sh/Analytics
No analytics have been recorded yet (nor will be during this `brew` run).
Installing from the API is now the default behaviour!
You can save space and time by running:
brew untap homebrew/core
==> Downloading https://formulae.brew.sh/api/formula_tap_migrations.jws.json
####################################################################################################################################################################################################################################### 100.0%
==> Auto-updated Homebrew!
Updated 5 taps (helm/tap, fairwindsops/tap, noobaa/noobaa, homebrew/core and homebrew/cask).
==> Fetching dependencies for noobaa/noobaa/noobaa: go
==> Fetching go
==> Downloading https://ghcr.io/v2/homebrew/core/go/manifests/1.20.7
####################################################################################################################################################################################################################################### 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/go/blobs/sha256:4a2d74187e6fa58781dd0fc4643ee847a3a2b5bffd103f048e532f53fd469d86
####################################################################################################################################################################################################################################### 100.0%
==> Installing noobaa/noobaa/noobaa dependency: go
==> Pouring go--1.20.7.x86_64_linux.bottle.tar.gz
๐บ /home/linuxbrew/.linuxbrew/Cellar/go/1.20.7: 11,997 files, 240.8MB
==> Fetching noobaa/noobaa/noobaa
==> Cloning https://github.com/noobaa/noobaa-operator.git
Cloning into '/home/jonathan/.cache/Homebrew/noobaa--git'...
==> Checking out tag v5.12.4
HEAD is now at ce3a871 Merge pull request #1145 from nimrod-becker/backport_to_5_12
==> Installing noobaa from noobaa/noobaa
==> go mod vendor
==> go generate
==> go build
๐บ /home/linuxbrew/.linuxbrew/Cellar/noobaa/5.12.4: 3 files, 66.0MB, built in 2 minutes 21 seconds
==> Running `brew cleanup noobaa`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
[jonathan@latitude ~]$ kubectl create ns noobaa
namespace/noobaa created
[jonathan@latitude ~]$ kubens noobaa
Context "microk8s" modified.
Active namespace is "noobaa".
[jonathan@latitude ~]$ noobaa install
INFO[0000] CLI version: 5.12.0
INFO[0000] noobaa-image: noobaa/noobaa-core:master-20220913
INFO[0000] operator-image: noobaa/noobaa-operator:5.12.0
INFO[0000] noobaa-db-image: centos/postgresql-12-centos7
INFO[0000] Namespace: noobaa
INFO[0000]
INFO[0000] CRD Create:
INFO[0000] โ
Already Exists: CustomResourceDefinition "noobaas.noobaa.io"
INFO[0000] โ
Already Exists: CustomResourceDefinition "backingstores.noobaa.io"
INFO[0000] โ
Already Exists: CustomResourceDefinition "namespacestores.noobaa.io"
INFO[0000] โ
Already Exists: CustomResourceDefinition "bucketclasses.noobaa.io"
INFO[0000] โ
Already Exists: CustomResourceDefinition "noobaaaccounts.noobaa.io"
INFO[0000] โ
Already Exists: CustomResourceDefinition "objectbucketclaims.objectbucket.io"
INFO[0000] โ
Already Exists: CustomResourceDefinition "objectbuckets.objectbucket.io"
INFO[0000]
INFO[0000] Operator Install:
INFO[0000] โ
Already Exists: Namespace "noobaa"
INFO[0000] โ
Created: ServiceAccount "noobaa"
INFO[0000] โ
Created: ServiceAccount "noobaa-endpoint"
INFO[0000] โ
Created: Role "noobaa"
INFO[0000] โ
Created: Role "noobaa-endpoint"
INFO[0000] โ
Created: RoleBinding "noobaa"
INFO[0000] โ
Created: RoleBinding "noobaa-endpoint"
INFO[0000] โ
Created: ClusterRole "noobaa.noobaa.io"
INFO[0000] โ
Created: ClusterRoleBinding "noobaa.noobaa.io"
INFO[0000] โ
Created: Deployment "noobaa-operator"
INFO[0000]
INFO[0000] System Create:
INFO[0000] โ
Already Exists: Namespace "noobaa"
INFO[0000] โ
Created: NooBaa "noobaa"
INFO[0000]
INFO[0000] NOTE:
INFO[0000] - This command has finished applying changes to the cluster.
INFO[0000] - From now on, it only loops and reads the status, to monitor the operator work.
INFO[0000] - You may Ctrl-C at any time to stop the loop and watch it manually.
INFO[0000]
INFO[0000] System Wait Ready:
INFO[0000] โณ System Phase is "". Deployment "noobaa-operator" is not ready: ReadyReplicas 0/1
INFO[0003] โณ System Phase is "". Pod "noobaa-operator-588b8fd64d-tpstb" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [noobaa-operator]). ContainersNotReady (containers with unready status: [noobaa-operator]).
INFO[0006] โณ System Phase is "". Pod "noobaa-operator-588b8fd64d-tpstb" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [noobaa-operator]). ContainersNotReady (containers with unready status: [noobaa-operator]).
INFO[0009] โณ System Phase is "". Pod "noobaa-operator-588b8fd64d-tpstb" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [noobaa-operator]). ContainersNotReady (containers with unready status: [noobaa-operator]).
INFO[0012] โณ System Phase is "". Pod "noobaa-operator-588b8fd64d-tpstb" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [noobaa-operator]). ContainersNotReady (containers with unready status: [noobaa-operator]).
INFO[0015] โณ System Phase is "". Pod "noobaa-operator-588b8fd64d-tpstb" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [noobaa-operator]). ContainersNotReady (containers with unready status: [noobaa-operator]).
INFO[0018] โณ System Phase is "". Pod "noobaa-operator-588b8fd64d-tpstb" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [noobaa-operator]). ContainersNotReady (containers with unready status: [noobaa-operator]).
INFO[0021] โณ System Phase is "". Pod "noobaa-operator-588b8fd64d-tpstb" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [noobaa-operator]). ContainersNotReady (containers with unready status: [noobaa-operator]).
INFO[0024] โณ System Phase is "". Pod "noobaa-operator-588b8fd64d-tpstb" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [noobaa-operator]). ContainersNotReady (containers with unready status: [noobaa-operator]).
INFO[0027] โณ System Phase is "". Pod "noobaa-operator-588b8fd64d-tpstb" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [noobaa-operator]). ContainersNotReady (containers with unready status: [noobaa-operator]).
^C
[jonathan@latitude ~]$ kubectl get po
NAME READY STATUS RESTARTS AGE
noobaa-operator-588b8fd64d-sq5gs 0/1 ErrImagePull 0 13s
[jonathan@latitude ~]$ kubectl describe po noobaa-operator-588b8fd64d-sq5gs
Name: noobaa-operator-588b8fd64d-sq5gs
Namespace: noobaa
Priority: 10000
Priority Class Name: normal-priority
Service Account: noobaa
Node: kube04/192.168.0.55
Start Time: Thu, 10 Aug 2023 21:02:50 +0100
Labels: app=noobaa
noobaa-operator=deployment
pod-template-hash=588b8fd64d
Annotations: cni.projectcalico.org/containerID: 4656999149f801a91618d43a1dc88408c9da8e7169c4387c19e0a0d53afcbb4e
cni.projectcalico.org/podIP: 10.1.102.112/32
cni.projectcalico.org/podIPs: 10.1.102.112/32
Status: Pending
SeccompProfile: RuntimeDefault
IP: 10.1.102.112
IPs:
IP: 10.1.102.112
Controlled By: ReplicaSet/noobaa-operator-588b8fd64d
Containers:
noobaa-operator:
Container ID:
Image: noobaa/noobaa-operator:5.12.0
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Limits:
cpu: 250m
memory: 512Mi
Requests:
cpu: 250m
memory: 512Mi
Environment:
OPERATOR_NAME: noobaa-operator
POD_NAME: noobaa-operator-588b8fd64d-sq5gs (v1:metadata.name)
WATCH_NAMESPACE: noobaa (v1:metadata.namespace)
NOOBAA_CLI_DEPLOYMENT: true
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lqnwt (ro)
/var/run/secrets/openshift/serviceaccount from oidc-token (rw)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
oidc-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3600
kube-api-access-lqnwt:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 100s default-scheduler Successfully assigned noobaa/noobaa-operator-588b8fd64d-sq5gs to kube04
Normal BackOff 21s (x4 over 94s) kubelet Back-off pulling image "noobaa/noobaa-operator:5.12.0"
Warning Failed 21s (x4 over 94s) kubelet Error: ImagePullBackOff
Normal Pulling 6s (x4 over 99s) kubelet Pulling image "noobaa/noobaa-operator:5.12.0"
Warning Failed 4s (x4 over 95s) kubelet Failed to pull image "noobaa/noobaa-operator:5.12.0": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/noobaa/noobaa-operator:5.12.0": failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:849ad0464b245e2b0ad6295e53efc80dae6b81532fbdfb609070975219969bea: not found
Warning Failed 4s (x4 over 95s) kubelet Error: ErrImagePull
I worked around by doing:
noobaa install --operator-image='noobaa/noobaa-operator:5.12.4' --noobaa-image='noobaa/noobaa-core:5.12.4'
Hi @djjudas21
Great workaround ๐
homebrew was updated to 5.14.2
Is this still happening?
Hi @djjudas21 Great workaround ๐ homebrew was updated to 5.14.2 Is this still happening?
5.14.4still have this question
Warning Failed 28s kubelet Failed to pull image "gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387": rpc error: code = Unknown desc = failed to pull and unpack image "gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387": failed to resolve reference "gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387": failed to do request: Head "https://gcr.io/v2/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar/manifests/v20221117-v0.1.0-22-g0e67387": dial tcp: lookup gcr.io on 1.1.1.1:53: read udp 1.1.1.1:51150->1.1.1.1:53: i/o timeout
Hi @djjudas21 Great workaround ๐ homebrew was updated to 5.14.2 Is this still happening?
5.14.4still have this question
Warning Failed 28s kubelet Failed to pull image "gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387": rpc error: code = Unknown desc = failed to pull and unpack image "gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387": failed to resolve reference "gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387": failed to do request: Head "https://gcr.io/v2/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar/manifests/v20221117-v0.1.0-22-g0e67387": dial tcp: lookup gcr.io on 1.1.1.1:53: read udp 1.1.1.1:51150->1.1.1.1:53: i/o timeout
openshift can install but kubenetes can nor