project-zot/zot

[Feat]: `HEAD /v2/$tag/blobs/$digest` could be faster

the-sun-will-rise-tomorrow opened this issue · 5 comments

the-sun-will-rise-tomorrow commented

Is your feature request related to a problem? Please describe.

HEAD /v2/$tag/blobs/$digest requests take an unusual amount of time. Even performing repeated requests with the same $digest is slow, 200-500ms every time.

Describe the solution you'd like

I am not sure why exactly HEAD /v2/$tag/blobs/$digest requests are slow, but if it's not possible to make the underlying operation itself faster, then an in-memory cache of the results would at least speed up repeated queries.

Describe alternatives you've considered

#2589, perhaps.

Additional context

I am experimenting with efficient uploading of images with many layers (built using Nix). In my experiments, the vast majority of the time seems to be spent on Skopeo asking Zot if a layer blob is already in the registry.

eusebiu-constantin-petu-dbk commented

Hello @the-sun-will-rise-tomorrow

Do you use authorization, can you post the config you are using?

Thank you!

the-sun-will-rise-tomorrow commented

Hi!

Here is the config: #2589 (comment)

Authorization is not needed for these HEAD requests in our case (anonymous read access).

eusebiu-constantin-petu-dbk commented

Hei @the-sun-will-rise-tomorrow I investigated this issue, and I put the profiler and run our benchmark tool to see what piece of code takes so much time.

The issue is bcrypt.CompareHashAndPassword([]byte(passphraseHash), []byte(passphrase)) which seems to be by design.
Check here: https://stackoverflow.com/questions/49437359/why-bcrypt-library-comparehashandpassword-method-is-slow
Maybe try this: https://security.stackexchange.com/questions/17207/recommended-of-rounds-for-bcrypt/83382#83382

Thanks!

the-sun-will-rise-tomorrow commented

Good insight, thank you. Maybe that function could be memoized (with the cache flushed every minute to avoid indefinitely retaining sensitive information in memory)? That should fix the performance problem in theory.