google/stenographer

multiple threads writing to same disk causes issues with disk cleanup

emillynge opened this issue · 10 comments

emillynge commented

Hi

I realize that having stenographer use multiple threads witing to different directories on the same underlying disk may not be a supported use case.

But I really do have a viable use for this.

In such a setup, the automatic disk cleanup does not work as intended since cleanup is done in serial per thread. The first thread to do cleanup will then take on the burden of cleaning files until the desired percentage of free space is achieved. Only if the first thread has no more files left to delete will other threads using the same disk begin deleting their files.

I think there are 2 viable solutions:

  1. quick and dirty - randomize the order in which threads calls SyncFiles(). This will work fairly well as long as the threads write equal amounts of data.

  2. more complex - move deletion to Env and delete files alternatingly.

I already have a fork with solution 1) that I could turn into a PR if you would like it.

gconnell commented

You're right that this is not an expected use-case, but you're also right that it's one we should probably support, since there's nothing saying you CAN'T do it. Given that, if you've got a PR you'd like to send out I'm happy to take a look.

emillynge commented

I can very quickly have a PR on solution 1) :)

Speaking of PRs, would you mind taking a look at #213 ? It has been open for about a month now and fixes a bug I found.

gconnell commented

Done, thanks, and sorry for the late reply!

rskjetlein commented

Was this fixed?

rskjetlein commented

I made a working solution for 1. mentioned. Can assemble a PR for it if any interest

jtgreen-cse commented

I'd be interested in seeing this merged. I've come across this issue and have this use case.

Edit: @rskjetlein I found your fork, here's the commit that fixes it -->rskjetlein@6f6d2cb

  • 😄1
rskjetlein commented

Will have a look. Its been dormant for some time...

rskjetlein commented

I'd be interested in seeing this merged. I've come across this issue and have this use case.

Edit: @rskjetlein I found your fork, here's the commit that fixes it -->rskjetlein@6f6d2cb

For the most part its working fine, during very high traffic rate (+10G) a unbalanced deletion can be observed.

jtgreen-cse commented

I've been testing the random thread picking strategy with 16 threads .. that makes it too random. Threads delete files very unevenly. Like suggested I think moving the logic in env and deciding which thread to pick for delete based on which has the most files may solve this.

rskjetlein commented

I've been testing the random thread picking strategy with 16 threads .. that makes it too random. Threads delete files very unevenly. Like suggested I think moving the logic in env and deciding which thread to pick for delete based on which has the most files may solve this.

I extended the API with a object delete function and had an external process that used the same API to control the disk fill percentage. The external process took into account other things that used the same disk, might be a usable approach for you.

Please reach out if you want to code snippets. rskjetlein@netrunner.nu