`FileCache`: better cleaning, free more than just the minimum
albertz opened this issue · 0 comments
albertz commented
Freeing exactly need_at_least_free_space_size
can be too little. E.g.:
- Multiple procs copy at the same time (e.g. RETURNN
FileCache
itself, see #1542, or other procs). After the space was freed, when the copying starts, or maybe during the copying, maybe some of the space was already taken again. - We actually need one more byte due to the current logic in
_copy_with_prealloc
. - I'm not sure if the filesystem maybe needs a few more bytes, e.g. for creating directories or so.
Another problem is that the current cleanup_files_wanted_older_than_days
default (7 days) is maybe way too high and is rarely used when it matters (to free enough space). Although, maybe some of the other suggestions below already are enough. Not sure.
Some suggestions:
- Fix
need_at_least_free_space_size = os.stat(src_filename).st_size + 1
. - Always try to free more than
need_at_least_free_space_size
, maybe sth like2 * need_at_least_free_space_size
or so. However, if this fails, it's only a warning. Only if we fail to freeneed_at_least_free_space_size
, it would be an error.
Other suggestions?