Integration build pushes image successfully using credentials but fails on pulling built image

[Nickmman]

What happened?

I have an IntegrationPlatform which is configured with a private registry, using a secret that contains a dockerauth. The build process works successfully. However, when running the Integration, which uses the same set of credentials, the image pull fails.

I have manually confirmed that the credentials can successfully pull the image.

The Integration is configured to use a service account (which has the imagePullSecrets configured), in which pulling the image still doesn't work. As per the documentation, when the Integration does not specify the imagePullSecrets/ServiceAccount to use, it will default to whatever is configured at the IntegrationPlatform level, which unfortunately does not work either.

Steps to reproduce

1. Create a secret (in this case named camel-dockerauth) which contains a dockerconfigjson
2. Deploy camel-k via helm with the following values:

  values:
    fullnameOverride: camel
    platform:
      build:
        registry:
          address: redacted.io
          organization: camel
          secret: camel-dockerauth

3. Deploy an Integration (in this case manually)
4. Watch as the Integration image is built but when deploying the image it does not succeed

Relevant log output

Failed to pull image "redacted.io/camel/camel-k-kit-csbbtv5qhr6c73at2ta0@sha256:efabd03db8c9df31e6b8e7454a47e2a7986cc6a7dc482533d2c2075c5494a642":
failed to pull and unpack image "redacted.io/camel/camel-k-kit-csbbtv5qhr6c73at2ta0@sha256:efabd03db8c9df31e6b8e7454a47e2a7986cc6a7dc482533d2c2075c5494a642":
failed to resolve reference "redacted.io/camel/camel-k-kit-csbbtv5qhr6c73at2ta0@sha256:efabd03db8c9df31e6b8e7454a47e2a7986cc6a7dc482533d2c2075c5494a642":
pull access denied, repository does not exist or may require authorization: authorization failed: no basic auth credentials

Camel K version

v1.12.1

[squakez]

That version is a bit old. I think the default in that version was the Spectrum strategy which we're not supporting any longer. You may try to change IntegrationPlatform and move to Jib publishing strategy instead. However that would be available in version 2.x only. Would you mind to try the very same configuration you have with a brand new 2.x installation? We can better figure it out which could be the root cause.

[Nickmman]

@squakez I'm deploying via the helm chart, from what I can see only version 0.13.1 (of the helm chart) is available

It looks like the version of the image of the operator changes, as well as the appVersion, but the version of the helm chart remains the same. It could be that this is where my issue lies, as no "newer" version of the helm chart is available, though the values do change.

I'll try updating the image of the operator via the helm chart values and get back to you.

Thanks!

[Nickmman]

@squakez I just deployed the operator with the newer container version. The integration is no longer being built anymore and the operator is complaining about missing CRDs. I do see that these CRDs exist in the helm chart but as I said in my previous comment, these are probably not being picked up due to the helm chart version still being 0.13.1.

[squakez]

Within version 2.0 we changed the charts to reflect the operator versioning. This is now to 2.4 and I can see the chart as well: https://artifacthub.io/packages/helm/camel-k/camel-k/2.4.0 - maybe you need to refresh the chart locally?

[Nickmman]

@squakez Ah, my apologies, I was looking at the Chart.yaml in GitHub only. I have redeployed using 2.4 and recreated the IntegrationPlatform. I can confirm that it is now using the Jib publishing strategy.

However, the original issue still persists. The image is built and pushed to the registry successfully but it fails to pull it back for running the Integration.

[squakez]

Thanks for testing. Then, it seems to be some kind of problem with the either Kubernetes or the registry itself. I suggest running two additional tests to verify where the problem happen.

1. Try to run the kit from an external Pod and using the private secret. You can follow the guidelines in https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#create-a-pod-that-uses-your-secret - use the image "redacted.io/camel/camel-k-kit-csbbtv5qhr6c73at2ta0@sha256:efabd03db8c9df31e6b8e7454a47e2a7986cc6a7dc482533d2c2075c5494a642".
2. Try to use an alternative registry just for the sake of testing. Most of our verification are done against Docker Hub using the pull secret configuration guide - having some other registry tested would be good to understand if the problem is on the registry side.

Please, reports the outcome so we can see how we can keep helping.

The real strange thing is that the same mechanism works fine with push but not with pull. You may also want to check any particular requirements needed on the registry (ie, the registry may be configured to push but not to pull?). For instance, I can see the GitHub package registry requires the user to configure either read or write operations: