Filestore cleanup is too slow
TheDiscordian opened this issue · 8 comments
Maybe we can speed it up by only removing offset 0 references, see if that fixes filestore errors just as well, and we can let the gc clean up the rest later.
It's specifically slow especially with large files, this would solve that aspect at least.
Note: On atrociously massive filesets, just the act of running filestore/verify results in an extreme wait time to just receive the list of blocks. Might need an upstream patch (an index on status, and a filter option on the HTTP API).
Instead of running filestore/verify across all blocks (like we do now), we can get the blocks we need by running ls on the CID of the object we need the first block of, until we get a returned size of 262144 or less I believe. Requires a bit of research, but this would pull up the effected blocks, and we could check them in filestore directly.
I want to try the offset 0 idea too still, as this could still be a tad slow for large files. If we do end up removing multiple blocks, do it in a batch for each file, so there won't be a brutal slowdown.
We can't remove just offset 0, we must remove all the blocks, but we can just check offset 0, before we build the list to remove (1 bad block, then they're all bad blocks EDIT: This is incorrect, some blocks might be shared by other objects).
filestore/verify can be asked to verify just a specific object. This will hopefully dump us back the blocks that are truly unfindable, and then we can safely remove those :D.
Unfortunately it doesn't take an object, but actually a block, this is a shame. We can use refs to get all the CIDs, can't omit the irrelevant ones, but that's fine for now.
Please haste, codefriend
Fixed in 78d6721