datastax/cass-config-builder pull error on OCP
Closed this issue · 11 comments
What happened?
When deploying the certified DataStax Kubernetes Operator for Apache Cassandra v1.14.0 on OCP 4.12.5, following issue has been reported:
Using default value for the CassandraDatacenter CR, the initContainer doesn't come up due to the container image not being found, with the error message Back-off pulling image "datastax/cass-config-builder:1.0-ubi7"
$ oc get po -n openshift-operators
development-dc1-rack1-sts-0 0/2
Init:ImagePullBackOff 0 19m
development-dc1-rack2-sts-0 0/2
Init:ImagePullBackOff 0 19m
development-dc1-rack3-sts-0 0/2
Init:ImagePullBackOff 0 19m
After some investigation, we found that the 1.0-ubi7 tag is not present on the registry.connect.redhat.com repo:
$ podman search --list-tags registry.connect.redhat.com/datastax/cass-config-builder
NAME TAG
registry.connect.redhat.com/datastax/cass-config-builder 1.0.0
registry.connect.redhat.com/datastax/cass-config-builder latest
registry.connect.redhat.com/datastax/cass-config-builder 1.0.0-2
registry.connect.redhat.com/datastax/cass-config-builder 1.0.3-ubi7-2
(It is however present on docker.io)
Could you update the registry.connect.redhat.com/datastax/cass-config-builder accordingly and push the 1.0-ubi7 tag there ?
What did you expect to happen?
No response
How can we reproduce it (as minimally and precisely as possible)?
See above
cass-operator version
v1.14.0
Kubernetes version
OCP 4.12.5
Method of installation
Operator Hub Certified Catalog
Anything else we need to know?
No response
I'll add it on the 1.16 release to automatically pull from registry (with the hash code).
@burmanm My understanding is that the datastax/cass-config-builder:1.0-ubi7 image is missing from redhat.registry.com and needs to be pushed there.
You're right, but currently we push releases from this repo when making one (and do the preflight check which is required to push there).
https://github.com/k8ssandra/cass-operator/blob/master/.github/workflows/release.yml#L121
The config-builder project does not have the ability to automatically push new versions there (or rebuild the existing ones). And as such, we might need to do it from this repo. And we also use the SHA (as recommended/required) in those builds:
https://github.com/k8ssandra/cass-operator/blob/master/scripts/release-certified-bundles.sh#L29
I'll try to see later if I have the preflight installed on any of my machines to check if I could manually push it meanwhile. All those older images were pushed when the process was different.
Understood. Please let me know if there's any help I can provide with the preflight tool or around certification in general.
Thanks for looking into this !
Hey, this should be available now:
When I ran docker pull, I received error.
% docker pull datastax/cass-config-builder:1.0-ubi7
1.0-ubi7: Pulling from datastax/cass-config-builder
no matching manifest for linux/arm64/v8 in the manifest list entries
There is no ubi7 arm64 image. You can use the 1.0 tag without suffix on arm
There is no ubi7 arm64 image. You can use the 1.0 tag without suffix on arm
Thanks for replying. I was following this https://docs.k8ssandra.io/install/local/single-cluster-helm/ , but ran into the errror
% kubectl get pods -n k8ssandra-operator
NAME READY STATUS RESTARTS AGE
demo-dc1-default-sts-0 0/2 Init:ImagePullBackOff 0 4m24s
demo-dc1-default-sts-1 0/2 Init:ImagePullBackOff 0 4m24s
demo-dc1-default-sts-2 0/2 Init:ImagePullBackOff 0 4m24s
k8ssandra-operator-86dc65b869-7zlhx 1/1 Running 0 6m8s
k8ssandra-operator-cass-operator-67875bf9fb-9b7rw 1/1 Running 0 6m8s
And then I double checked the status
Warning Failed 2m43s (x4 over 4m14s) kubelet Failed to pull image "datastax/cass-config-builder:1.0-ubi7": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/datastax/cass-config-builder:1.0-ubi7": no match for platform in manifest: not found
In this case, how should I resolve it? override the chart k8ssandra https://helm.k8ssandra.io/stable with something else?
When using arm, you need to override the config-builder image. There's a ConfigMap that's deployed (ends with -manager-config) and there's a part:
image_config.yaml: |
apiVersion: config.k8ssandra.io/v1beta1
kind: ImageConfig
images:
system-logger: "k8ssandra/system-logger:v1.16.0"
config-builder: "datastax/cass-config-builder:1.0-ubi7Change the config-builder to one without the -ubi7 and restart cass-operator (kubectl rollout restart deployment ..)
@burmanm Thanks for answering me. I think there is still a bug k8ssandra/k8ssandra-operator#1084 that makes me fail to spin up the stargate service .