Is OCI-based image not yet supported by docker private registry?
tonyliu666 opened this issue · 11 comments
Description
I would like to push the oci-based image to private docker registry thorugh buildah cli, but I found a log said that DEBU[0000] Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json]. Does it mean currently private docker registry only support docker media type? Following are the logs:
Reproduce
buildah push --creds=xxx --tls-verify=false --format docker --debug localhost/redddis:latest 192.168.56.4:30010/redddis:latest
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] cached value indicated that overlay is supported
DEBU[0000] cached value indicated that metacopy is being used
DEBU[0000] cached value indicated that native-diff is not being used
INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true
DEBU[0000] Assuming docker:// as the transport method for DESTINATION: docker://192.168.56.4:30010/redddis:latest
DEBU[0000] Looking up image "localhost/redddis:latest" in local containers storage
DEBU[0000] Trying "localhost/redddis:latest" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@915bdb79316c2af4c5fe34d877a8b33c765346c3745e211881566fa5ffd231a3"
DEBU[0000] Found image "localhost/redddis:latest" as "localhost/redddis:latest" in local containers storage
DEBU[0000] Found image "localhost/redddis:latest" as "localhost/redddis:latest" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@915bdb79316c2af4c5fe34d877a8b33c765346c3745e211881566fa5ffd231a3)
DEBU[0000] Pushing image localhost/redddis:latest to docker://192.168.56.4:30010/redddis:latest
DEBU[0000] Copying source image [overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@915bdb79316c2af4c5fe34d877a8b33c765346c3745e211881566fa5ffd231a3 to destination image //192.168.56.4:30010/redddis:latest
DEBU[0000] Returning credentials for 192.168.56.4:30010 from DockerAuthConfig
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration
DEBU[0000] Using "default-docker" configuration
DEBU[0000] Using file:///var/lib/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/192.168.56.4:30010
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf"
DEBU[0000] Using blob info cache at /var/lib/containers/cache/blob-info-cache-v1.boltdb
DEBU[0000] IsRunningImageAllowed for image containers-storage:[overlay@/var/lib/containers/storage]@915bdb79316c2af4c5fe34d877a8b33c765346c3745e211881566fa5ffd231a3
DEBU[0000] Using transport "containers-storage" policy section ""
DEBU[0000] Requirement 0: allowed
DEBU[0000] Overall: allowed
Getting image source signatures
DEBU[0000] Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json]
DEBU[0000] Checking /v2/redddis/blobs/sha256:291f5b7439a09f8f7666c9a2d95b55176aecd6711a92b4a83d3285c26be13d32
DEBU[0000] GET https://192.168.56.4:30010/v2/
DEBU[0000] Ping https://192.168.56.4:30010/v2/ err Get "https://192.168.56.4:30010/v2/": http: server gave HTTP response to HTTPS client (&url.Error{Op:"Get", URL:"https://192.168.56.4:30010/v2/", Err:(*errors.errorString)(0xc0002f21d0)})
DEBU[0000] GET http://192.168.56.4:30010/v2/
DEBU[0000] Ping http://192.168.56.4:30010/v2/ status 200
DEBU[0000] HEAD http://192.168.56.4:30010/v2/redddis/blobs/sha256:291f5b7439a09f8f7666c9a2d95b55176aecd6711a92b4a83d3285c26be13d32
DEBU[0000] ... not present
DEBU[0000] exporting filesystem layer "291f5b7439a09f8f7666c9a2d95b55176aecd6711a92b4a83d3285c26be13d32" without compression for blob "sha256:291f5b7439a09f8f7666c9a2d95b55176aecd6711a92b4a83d3285c26be13d32"
DEBU[0000] No compression detected
DEBU[0000] Compressing blob on the fly
DEBU[0000] Uploading /v2/redddis/blobs/uploads/
DEBU[0000] POST http://192.168.56.4:30010/v2/redddis/blobs/uploads/
Copying blob 291f5b7439a0 [--------------------------------------] 8.0b / 2.7MiB
DEBU[0000] PATCH http://192.168.56.4:30010/v2/redddis/blobs/uploads/e50b2c3d-860f-4f0f-8cd4-9dbecbd71c37?_state=omxMkVVWSni4EZ156g4ax2axx-Jz147gB9oS5npNtoB7Ik5hbWUiOiJyZWRkZGlzIiwiVVVJRCI6ImU1MGIyYzNkLTg2MGYtNGYwZi04Y2Q0LTlkYmVjYmQ3MWMzNyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyNC0wNS0zMFQxNjo
Copying blob 291f5b7439a0 done
DEBU[0000] PUT http://192.168.56.4:30010/v2/redddis/blobs/uploads/e50b2c3d-860f-4f0f-8cd4-9dbecbd71c37?_state=hM8t_Cfx4AZFsZi_9lAO1eQR3iVKc7fNROhs1Uay3Q57Ik5hbWUiOiJyZWRkZGlzIiwiVVVJRCI6ImU1MGIyYzNkLTg2MGYtNGYwZi04Y2Q0LTlkYmVjYmQ3MWMzNyIsIk9mZnNldCI6MzE4NjAzLCJTdGFydGVkQXQiOiIyMDI0LTA1LTMwVDE2OjA4OjI1WiJ9&digest=sha256%3Ab4955b533a5217e4ec86bd605a7331e08ae404b6f9d119e35b1728e466fa9ca6
DEBU[0000] Upload of layer sha256:b4955b533a5217e4ec86bd605a7331e08ae404b6f9d119e35b1728e466fa9ca6 complete
DEBU[0000] exporting opaque data as blob "sha256:915bdb79316c2af4c5fe34d877a8b33c765346c3745e211881566fa5ffd231a3"
DEBU[0000] No compression detected
DEBU[0000] Using original blob without modification
DEBU[0000] Checking /v2/redddis/blobs/sha256:915bdb79316c2af4c5fe34d877a8b33c765346c3745e211881566fa5ffd231a3
DEBU[0000] HEAD http://192.168.56.4:30010/v2/redddis/blobs/sha256:915bdb79316c2af4c5fe34d877a8b33c765346c3745e211881566fa5ffd231a3
DEBU[0000] ... not present
DEBU[0000] Uploading /v2/redddis/blobs/uploads/
DEBU[0000] POST http://192.168.56.4:30010/v2/redddis/blobs/uploads/
DEBU[0000] PATCH http://192.168.56.4:30010/v2/redddis/blobs/uploads/d5960176-6c79-4f21-88a8-1c2d6482953c?_state=cNANyxSfoujqMLSwdWcUT2ecwW-SwZs-svgV5JDBGA17Ik5hbWUiOiJyZWRkZGlzIiwiVVVJRCI6ImQ1OTYwMTc2LTZjNzktNGYyMS04OGE4LTFjMmQ2NDgyOTUzYyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyNC0wNS0zMFQxNjowODoyNS43NTgwMjYxODRaIn0%3D
Copying config 915bdb7931 done
DEBU[0000] PUT http://192.168.56.4:30010/v2/redddis/blobs/uploads/d5960176-6c79-4f21-88a8-1c2d6482953c?_state=J7EUerEtrukM_e7yq-dZpuccOnfhw6-aqpQOUofK_e57Ik5hbWUiOiJyZWRkZGlzIiwiVVVJRCI6ImQ1OTYwMTc2LTZjNzktNGYyMS04OGE4LTFjMmQ2NDgyOTUzYyIsIk9mZnNldCI6MzI2LCJTdGFydGVkQXQiOiIyMDI0LTA1LTMwVDE2O
Copying config 915bdb7931 done
DEBU[0001] Upload of layer sha256:915bdb79316c2af4c5fe34d877a8b33c765346c3745e211881566fa5ffd231a3 complete
Writing manifest to image destination
DEBU[0001] PUT http://192.168.56.4:30010/v2/redddis/manifests/latest
Storing signatures
DEBU[0001] pushed image "192.168.56.4:30010/redddis:latest@sha256:413baedd0dea0496bc0062e85c49b0021afa4abe57662a124bc7886c673e6ecb" with digest sha256:413baedd0dea0496bc0062e85c49b0021afa4abe57662a124bc7886c673e6ecb
DEBU[0001] Successfully pushed docker://192.168.56.4:30010/redddis:latest with digest sha256:413baedd0dea0496bc0062e85c49b0021afa4abe57662a124bc7886c673e6ecb
DEBU[0001] shutting down the store
Expected behavior
push the oci-based image to registry successfully.
registry version
registry github.com/docker/distribution 2.8.3
Additional Info
I run docker registry as a daemonset on k8s.
This registry is OCI compliant/conformant. I dont know what buildah does so I can't possibly comment on that
That's weird. I use skopeo cli to inspect the image format in the private docker registry. That is exactly the same as oci format.
skopeo inspect --creds myuser:mypasswd --tls-verify=false --raw docker://192.168.56.3:30010/rdis:latest | jq
{
"schemaVersion": 2,
"config": {
"mediaType": "application/vnd.oci.image.config.v1+json",
"digest": "sha256:3fcb806294acf28daf058e4c55b9e2a3d89deeaca3400e9b9e40c5710c3f6706",
"size": 326
},
"layers": [
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:e76f7e7e75dc91c3bfc97cfa55e64978c72a1e17a1c324d4abac626494012195",
"size": 318639
}
],
"annotations": {
"io.kubernetes.cri-o.annotations.checkpoint.name": "rdis1"
}
}
I still don't know what the problem here is.
But I did notice you are using the release that's basically in a maintenance mode. Please use the latest v3 release
I briefly talk about what I am doing. I just try to follow the steps mentioned in this link, https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/#restore-checkpointed-container-k8s. But as the logs I posted above when buildah is trying to push the layer containing the checkpointed data, it would fail in the middle of the process. The corresponding log is like this: Copying blob 291f5b7439a0 [--------------------------------------] 8.0b / 2.7MiB. If you have no idea about this, it doesn't matter and then I can close this issue.
If you have no idea about this, it doesn't matter and then I can close this issue.
I think I have an idea or two about this project 😄 I'm trying to understand the problem you are reporting so I can help you. That's all.
You said this in the original comment:
I found a log said that DEBU[0000] Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json].
Sorry, I'm not familiar with every tool in the container ecosystem so by proxy I'm obviously not familiar with the logs these tools produce and find it hard to "parse" them by my eyes.
What I would like to hear taking into account your latest comment:
"I'm pushing an image with a tool called buildah and the push fails when a blob is uploaded. I suspect it's because registry isn't OCI compliant. Here's why..."
Instead, I'm seeing some manifest and I don't know what is its relation to the problem. Is buildah producing confusing logs? I don't know what "candidate list" is....or what buildah means by "candidate list" 🤷♂️
Thanks for your help. After updating the registry version to the latest one, I found that the original manifest logs become:
DEBU[0000] Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v1+json]
I think I have updated the registry that is OCI compliant, but the debugging logs still say the uploaded blob is unkown. These are partial logs related with this:
DEBU[0000] PATCH http://192.168.56.4:30010/v2/red1s/blobs/uploads/34232514-b2ff-4246-8e5e-fea10d85c917?_state=pQnBs7tmI_XRPLHLe_t0cK6PEiSbdIRTYjMNXgGO6hN7Ik5hbWUiOiJyZWQxcyIsIlVVSUQiOiIzNDIzMjUxNC1iMmZmLTQyNDYtOGU1ZS1mZWExMGQ4NWM5MTciLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjQtMDYtMDJUMTQ6MjQ6MDQuNTc4NTA2OTQ4WiJ9
Copying blob 2333ef4faf9f done
WARN[0000] failed, retrying in 2s ... (1/3). Error: writing blob: uploading layer chunked: blob upload unknown: blob upload unknown to registry
I think the functionality of buildah is same as docker cli. Because right now I just know how to use this instead of knowing the whole details about buildah, I am not sure about buildah will produce some confusing logs either. And as for candidate list, buildah probably have few options that it can specify which image format that it is able to send image to registry.
I think the functionality of buildah is same as docker cli
Can you reproduce this issue with docker cli?
The error you are seeing is the registry can not find a blob upload that matches the upload UUID and state...no idea how buildah work. I'd recommend asking them
This issue has become a little confused.
The original comman has the --format docker flag, which tells buildah to not use the OCI format, which would explain the issue in the title.
For the problem with the layers, I expect the best way to provide a reproduce is to provide the buildah commands you used to construct the image before attempting the push, and also, try running it on a new registry without any existing storage.
Sorry for late reply. @milosgajdos If I use docker cli, I need to transfer the oci format to docker format at first and then push the image. Here are my steps:
- buildah push --format oci localhost/red1s:latest docker-archive:/home/vagrant/redis-oci.tar 2. docker import /home/vagrant/redis-oci.tar red1s:latest 3. docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
192.168.56.4:30010/red1s latest dd0aeb6c23e4 2 days ago 2.94MB
4.docker push 192.168.56.4:30010/red1s:latest
The push refers to repository [192.168.56.4:30010/red1s]
2333ef4faf9f: Pushing [==================================================>] 2.974MB
unknown blob
The same problem exists here. @Jamstah without using the --format docker flag, the issue is still there.
buildah push --creds=xxx --tls-verify=false --debug localhost/red1s:latest 192.168.56.4:30010/red1s:latest
WARN[0002] failed, retrying in 2s ... (2/3). Error: writing blob: uploading layer to http://192.168.56.4:30010/v2/red1s/blobs/uploads/973be251-2b59-4556-9d6d-4958152cf9e0?_state=i5EWxeGvoeqY4rO5nbTbdVXh-OlLblLPkti5eV4eGJZ7Ik5hbWUiOiJyZWQxcyIsIlVVSUQiOiI5NzNiZTI1MS0yYjU5LTQ1NTYtOWQ2ZC00OTU4MTUyY2Y5ZTAiLCJPZmZzZXQiOjMyNiwiU3RhcnRlZEF0IjoiMjAyNC0wNi0wNVQwMzo0Mzo1NVoifQ%3D%3D&digest=sha256%3Add0aeb6c23e4db41100a5fc1972b24caec994620836636a424eecb29d2365648: blob upload unknown: blob upload unknown to registry
And the commands I created the image before pushing to docker registry:
- newcontainer=$(buildah from scratch)
- buildah add $newcontainer /var/lib/kubelet/checkpoints/checkpoint-_--.tar /
- buildah config --annotation=io.kubernetes.cri-o.annotations.checkpoint.name= $newcontainer
- buildah commit $newcontainer checkpoint-image:latest
- buildah rm $newcontainer
I think maybe the issue comes from the overlaying storage. The storage I used for registries is deployed a nfs server on another VM and the three docker private registries pod on different nodes(Daemonset) connect with it. (NFS use single share folder with three different private registry pods). I will try to distribute one single nfs folder to each registry pod instead of sharing the same folder.
After creating a new test for distributing one single nfs folder to each registry pod, I found the error exists. Then I change the way to docker registry without existing storage, and it finally can succesfully push to docker registry. I don't know why the nfs backing storage will cause this issue maybe the network is unstable.
NFS can have some fun locking behaviours so I wouldn't be surprised.