sonatype-nexus-community/nexus-blobstore-google-cloud

Hard limit of Delete can cause problems with some cases

robertcsakany opened this issue · 5 comments

robertcsakany commented

I would like to campact blobs. Which works fine, only I have that problem that the chunk size on compact only 1000.

Example:

image

I've deleted with task from judong-snaphsot around 140 000 entries (blobs). The problem is the status only shows 1000 elements are deleted:

image

I've made a task which compact blob store in every 15 min, but several week can be while all deleted blob can be removed.

2021-11-25 22:02:45,429+0000 INFO  [quartz-10-thread-17]  *SYSTEM org.sonatype.nexus.blobstore.gcloud.internal.GoogleCloudBlobStore - ---- Elapsed time: 2.752 min, processed: 1000 ----
2021-11-25 22:02:45,430+0000 INFO  [quartz-10-thread-17]  *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Compact nexus-blobstore-judong-snapshots' [blobstore.compact] state change RUNNING -> WAITING (OK)
2021-11-25 22:02:45,433+0000 INFO  [quartz-10-thread-18]  *SYSTEM org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask - Task log: /nexus-data/log/tasks/blobstore.compact-20211125220245432.log
2021-11-25 22:02:45,433+0000 INFO  [quartz-10-thread-18]  *SYSTEM org.sonatype.nexus.blobstore.gcloud.internal.GoogleCloudBlobStore - Begin deleted blobs processing

As I see it's harcoded limit.

https://github.com/sonatype-nexus-community/nexus-blobstore-google-cloud/blob/main/nexus-blobstore-google-cloud/src/main/java/org/sonatype/nexus/blobstore/gcloud/internal/DeletedBlobIndex.java

Is there any way to make it a config? Because I understand the impact when it's too large, but sometimes for maintaining some repos required to increase that limit temporarly.

nblair commented

Thanks for the report, my apologies for the delay in response. That 1000 constant you are referencing isn't a limit per se, it's just the indicator when that warning is raised.

Google Cloud Datastore has a limit of 500 on the number of entities that can be referenced within a single transaction (see https://cloud.google.com/datastore/docs/concepts/limits for that and more). https://github.com/sonatype-nexus-community/nexus-blobstore-google-cloud/blob/main/nexus-blobstore-google-cloud/src/main/java/org/sonatype/nexus/blobstore/gcloud/internal/DeletedBlobIndex.java#L119 attempts to account for this by partitioning the list into groups of 500, then issuing deletes in serial. I admit there isn't a lot of observability in that code.

I'll try to refactor that code for some more logging. What version of the blobstore plugin and NXRM are you using? That will help me make a patched build you can try to get some more diagnostic information.

robertcsakany commented

Thanks. If you need any help count on me. I don't know nexus internals / practices, but I have experience on karaf / osgi.

nblair commented

Just opened #108 as a proposal. After some experimentation, it's unnecessary for that limit to be so low by default. I've increased that limit to 100,000 entries; that's still going to take quite a bit of time to execute on the storage size, but should allow each compaction task to get more work done.

nblair commented

I just merged the change and published 0.36.1 of the plugin. Let me know how your experience goes with the default value for nexus.gcs.deletedBlobIndex.contentQueryLimit. If you do end up changing it, I'd be curious to see what differences you observe.

Github automatically closed the issue upon the merge, please feel free to re-open if the situation doesn't get any better.