Garbage collection cronjob leaves registry in an inconsistent state

[0xDEC0DE]

Prerequisites

• kubectl
• skopeo
• A deployed registry configured with persistence.deleteEnabled = true and garbageCollect.enabled = true

Steps to reproduce

Setup

export REGISTRY=$NAME_OF_REGISTRY_INGRESS
export REGISTRY_POD=$NAME_OF_REGISTRY_POD_FROM_KUBECTL

docker login $REGISTRY
docker pull hello-world:latest
docker tag hello-world:latest ${REGISTRY}/hello-world:latest
docker push ${REGISTRY}/hello-world:latest
skopeo delete docker://${REGISTRY}/hello-world:latest
kubectl exec $REGISTRY_POD -- /bin/registry garbage-collect --delete-untagged=true /etc/docker/registry/config.yml

Test

docker push ${REGISTRY}/hello-world:latest
docker pull ${REGISTRY}/hello-world:latest

Expected result

Success

Actual behavior

It fails, claiming layers already exist, etc.

Workaround

Restarting the registry after garbage-collection makes it work as expected:

kubectl delete pod $REGISTRY_POD ; sleep 5  # XXX: restart registry
docker push ${REGISTRY}/hello-world:latest
docker pull ${REGISTRY}/hello-world:latest  # XXX: success

[0xDEC0DE]

Workaround

Disabling caching on the registry server will also make it behave correctly. This can be achieved by setting storage.cache.blobdescriptor to an "invalid" value in the configs, e.g.:

docker-registry:
  configData:
    storage:
      cache:
        blobdescriptor: disabled

[nabaruns]

Facing same issue.