Performance on large Registry with NFS

[WTFKr0]

Hi,

We use this script today and have to modify it for performance issue
We have a Registry on NFS, with ~10000 blobs and ~200 GB data
Your script tell us we have ~2000 MANIFESTS_WITHOUT_TAGS

The problem is that the find command (run on each manifest) take about 30 seconds :D so the clean will take about 17 hours
We modify the script for caching the find result in a file, and then use it :

  counter=1
  find . > /tmp/find.txt
  for manifest in ${MANIFESTS_WITHOUT_TAGS}; do
    echo "## Doing $counter / ${TOTAL_COUNT}"
    let counter=$counter+1
    repos=$(grep "_manifests/revisions/sha256/${manifest}/link" /tmp/find.txt | awk -F "_manifest"  '{print $(NF-1)}' | sed 's#^./\(.*\)/#\1#')

We add a counter too
With that, we clean our registry in 3 minutes

Is this can be done is your image ?
I can open a PR for that

[mortensteenrasmussen]

Well, that certainly seems like an optimization ;)

Please do open a PR, and I'll merge and create a new version of the image ASAP. And thanks!

[mortensteenrasmussen]

Thanks a bunch for the PRs. New release incoming in a few mins. Could you please test the new image and verify that it works? My test system is temporarily out of order ;)

[mortensteenrasmussen]

A new image is available: mortensrasmussen/docker-registry-manifest-cleanup:1.0.4

https://hub.docker.com/r/mortensrasmussen/docker-registry-manifest-cleanup/tags/

[WTFKr0]

I just tested, and it works as expected, thanx !