Image cannot be pulled due to unauthorized with private container registry
Closed this issue · 6 comments
Issue
OS: Macos
K8s cluster version: 1.21
When we start a build then the following error is reported as kubernetes event within the Build pod created
Describe(demo/buildpack-quarkus-buildrun-1-5zzzb-pod-8v64x) ─────────
Normal Pulling 5m4s (x3 over 5m48s) kubelet Pulling image "registry.local:5000/redhat-builder/quarkus"
Warning Failed 5m4s (x3 over 5m48s) kubelet Failed to pull image "registry.local:5000/redhat-builder/quarkus": rpc error:
code = Unknown desc = failed to pull and unpack image "registry.local:5000/redhat-builder/quarkus:latest":
failed to resolve reference "registry.local:5000/redhat-builder/quarkus:latest":
unexpected status code [maniests latest[]: 401 Unauthorized
Normal BackOff 25s (x23 over 5m47s) kubelet Back-off pulling image "registry.local:5000/redhat-builder/quarkus"
K8s files
apiVersion: shipwright.io/v1alpha1
kind: ClusterBuildStrategy
metadata:
name: quarkus-buildpacks
spec:
buildSteps:
- name: prepare
image: docker.io/paketobuildpacks/builder:full
imagePullPolicy: Always
securityContext:
runAsUser: 0
capabilities:
add:
- CHOWN
command:
- chown
args:
- -R
- "1000:1000"
- /tekton/home
resources:
limits:
cpu: 500m
memory: 1Gi
requests:
cpu: 250m
memory: 65Mi
- name: build-and-push
image: registry.local:5000/redhat-builder/quarkus
imagePullPolicy: Always
securityContext:
runAsUser: 1000
runAsGroup: 1000
command:
- /bin/bash
args:
- -c
- |
set -euo pipefail
mkdir /tmp/cache /tmp/layers
/cnb/lifecycle/creator \
'-app=$(params.shp-source-context)' \
-cache-dir=/tmp/cache \
-layers=/tmp/layers \
-report=/tmp/report.toml \
'$(params.shp-output-image)'
# Store the image digest
grep digest /tmp/report.toml | tr -d ' \"\n' | sed s/digest=// > "$(results.shp-image-digest.path)"
resources:
limits:
cpu: 500m
memory: 1Gi
requests:
cpu: 250m
memory: 65Mi
---
apiVersion: shipwright.io/v1alpha1
kind: Build
metadata:
name: buildpack-quarkus-build
namespace: demo
spec:
source:
url: https://github.com/snowdrop/quarkus-tap-petclinic
contextDir: .
strategy:
name: quarkus-buildpacks
kind: ClusterBuildStrategy
output:
image: registry.local:5000/quarkus-petclinic
credentials:
name: registry-creds
---
apiVersion: shipwright.io/v1alpha1
kind: BuildRun
metadata:
name: buildpack-quarkus-buildrun-1
namespace: demo
spec:
buildRef:
name: buildpack-quarkus-build
Creds to access the local registry have been created
REGISTRY_HOST="registry.local:5000" REGISTRY_USER=admin REGISTRY_PASSWORD=snowdrop
kubectl create secret docker-registry registry-creds -n demo \
--docker-server="${REGISTRY_HOST}" \
--docker-username="${REGISTRY_USER}" \
--docker-password="${REGISTRY_PASSWORD}"
I guess we lack at least some documentation on this, but to analyze:
- To pull private images, one needs an imagePullSecret.
- ImagePullSecrets can be defined at different places:
- On the pod, that's not under the control of the build strategy author
- On the ServiceAccount that is referenced in the BuildRun, that's under the control of the build user, but that user does not necessarily know which images a build strategy needs
- In the Kubernetes node.
- What complicates the situation is that we support ClusterBuildStrategies which therefore can reference an image in a cluster-scoped resource, but secrets (and service accounts) are always namespaced. For the namespaced build strategy, we could probably easily add an
imagePullSecretsfield that we propagate to the Pod.
Actually, in my local cluster for my own stuff, I use a namespace build strategy and have the imagePullSecret defined in the service account that all BuildRuns are using.
In Code Engine, we have the secret in the node.
Tekton's ClusterTasks probably face the same issue, but I have never checked if they have creative ideas for this problem.
To immediately help you @cmoulliard. You seem to be working in a namespace called demo. You could add the secret to the default ServiceAccount as imagePullSecret, as you don't specify any ServiceAccount information in the BuildRun.
To immediately help you @cmoulliard. You seem to be working in a namespace called
demo. You could add the secret to the default ServiceAccount as imagePullSecret, as you don't specify any ServiceAccount information in the BuildRun.
Thanks. will do that and try :-)
Actually, in my local cluster for my own stuff, I use a namespace build strategy and have the imagePullSecret defined in the service account that all BuildRuns are using.
Can you show me with an example where you specify that ? @SaschaSchwarze0
Can you show me with an example where you specify that ? @SaschaSchwarze0
Fix using -->
apiVersion: shipwright.io/v1alpha1
kind: BuildRun
metadata:
name: buildpack-quarkus-buildrun-1
namespace: demo
spec:
serviceAccount:
name: registry-creds
generate: false
buildRef:
name: buildpack-quarkus-build
My idea was that you continue to NOT specify a ServiceAccount in the BuildRun and then do the following in the default Service Account of your namespace:
apiVersion: v1
kind: ServiceAccount
metadata:
name: default
namespace: demo
imagePullSecrets:
- name: registry-credsAlternatively you can also create another ServiceAccount where you reference the imagePullSecret, but you'll then have to reference it in all BuildRuns.
Alternatively you can also create another ServiceAccount where you reference the imagePullSecret, but you'll then have to reference it in all BuildRuns.
This is what I did and that works