Signatures do not match across backing registries
BenTheElder opened this issue · 5 comments
What happened:
See:
kubernetes/registry.k8s.io#187 and https://kubernetes.slack.com/archives/CJH2GBF7Y/p1679166550351119
Images should have identical digests no matter what region I pull from.
This does not appear to be the case for some of the sigstore images added by the image-promoter
What you expected to happen:
Images should be identical in all backing registries
How to reproduce it (as minimally and precisely as possible):
Check us-west1 vs us-west2 AR instances for provider-aws/aws-ebs-csi-driver:sha256-c75878156614efc7c501ea655cd9da1ede35e9aee252436a92ff01f67f1c53fa.sig
Anything else we need to know?:
Environment:
- Cloud provider or hardware configuration:
- OS (e.g:
cat /etc/os-release): - Kernel (e.g.
uname -a): - Others:
See: kubernetes/registry.k8s.io#187 (comment)
I think this is relatively unlikely to break anyone as long as it remains scoped to sigstore signatures (which appears to be the case) given the usage patterns for this type of """image""". Still worth fixing.
Looks like this is also "some signatures are missing in some regions" kubernetes/registry.k8s.io#187 (comment)
xref: kubernetes/release#2962
The signature oci objects are diverging because the promoter died mid process. This PR will not fix the rate limit killing kpromo but at least will stop the propagation of new divergent .sig "images": #809
Remediation is being tracked in kubernetes/release#2962
Thanks!
Re: killed by rate-limit – We should also consider retry with-backoff on error in case of e.g. network flakes. IIRC crane exposes an API for this and crane copy does this by default.