spegel fails to trigger pulling new images

[gsimko]

Spegel version

v0.22.0

Kubernetes distribution

k3s

Kubernetes version

v1.29.0

CNI

flannel

Describe the bug

My spegel setup is failing on me. How could I debug what's happening?

I get an error 404 if I try to pull the manifest directly from the server running k8s:
curl -v "http://localhost:30021/v2/isindir/sops-secrets-operator/manifests/0.12.3?ns=docker.io"

However, if I pull the image with crictl beforehand, the curl command above suddenly starts working:
k3s crictl pull docker.io/isindir/sops-secrets-operator:0.12.3

That suggests me that spegel won't trigger pulling the image, only tries to look up the existing ones. That is not the expected behavior, right?

My hosts.toml for docker.io is the following:

server = 'https://registry-1.docker.io'

[host]
[host.'http://MYIP:30020']
capabilities = ['pull', 'resolve']

[host.'http://MYIP:30021']
capabilities = ['pull', 'resolve']

In the spegel logs this is what I see:

{"time":"2024-05-31T00:33:33.002503766Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handleMirror","file":"/build/pkg/registry/registry.go","line":236},"msg":"handling mirror request from external node","key":"docker.io/isindir/sops-secrets-operator:0.12.3","path":"/v2/isindir/sops-secrets-operator/manifests/0.12.3","ip":"10.42.1.1"}
{"time":"2024-05-31T00:33:33.006250663Z","level":"ERROR","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":135},"msg":"","err":"mirror resolve retries exhausted for key: docker.io/isindir/sops-secrets-operator:0.12.3","path":"/v2/isindir/sops-secrets-operator/manifests/0.12.3","status":404,"method":"HEAD","latency":"3.795709ms","ip":"10.42.1.1"}

Do you have any suggestion what is going wrong? Thanks!

[bittrance]

That is exactly how spegel is supposed to work, with one critical caveat. You can rerun your experiment, but crictl pull the image on another node, and spegel should still find it.

Put differently, spegel is not a pull-through cache, but rather a side-car that distributes the existing containerd stores between nodes. It will never initiate communication with external repositories, because containerd is eminently capable of doing that on its own.

spegel operates as a containerd mirror and containerd reacts to the spegel 404 by moving on to the next mirror, eventually falling back to its internal image pulling. Hence, from from an external observer's perspective the spegel 404 is not visible.

[phillebaba]

I think that the explanation covers most of it.

Closing as this is not a bug but rather expected behavior.

[gsimko]

@phillebaba that makes sense, thank you for the answer! I would suggest putting that information ("spegel is not a pull-through cache, but rather a side-car") in the main README.md to make it obvious what to expect.