potential race condition prevents 'unix.rm' from deleting directory
Opened this issue · 2 comments
Raised in #205 by @scott314159, there seems to be a race condition during the finalization command of the Pyaflowa preprocessing module where the main job tries to delete the log file directory but returns OSError: [Errno 39] Directory not empty , which traces back from shutil.rmtree
.
This seems to be a know issue with shutil.rmtree
, see e.g., https://stackoverflow.com/questions/11228079/python-remove-directory-error-file-exists that happens on NFS file systems where file locks are present.
My thinking is that this is a race condition where the file locks cannot be removed before the rm
command tries to delete. One potential try is to put a try-except clause in unix.rm
that waits a few seconds if it hits this OSError, to give the filesystem some time to remove it's lock.
seisflows/seisflows/tools/unix.py
Lines 197 to 205 in d7661ee
Hey @scott314159, again, sorry for the long pause in communication here! Yeah the log message removal is intended to reduce file count, as for large-scale problems there may be (hundreds of) thousands of files created which may stress file systems, but I agree there is usually useful information there. The parameter export_log_files
is meant to make it a User decision to save these files by copying them to a more permanent location, and is set True by default.