adjtomo/seisflows

potential race condition prevents 'unix.rm' from deleting directory

Opened this issue · 2 comments

bch0w commented

Raised in #205 by @scott314159, there seems to be a race condition during the finalization command of the Pyaflowa preprocessing module where the main job tries to delete the log file directory but returns OSError: [Errno 39] Directory not empty , which traces back from shutil.rmtree.

This seems to be a know issue with shutil.rmtree, see e.g., https://stackoverflow.com/questions/11228079/python-remove-directory-error-file-exists that happens on NFS file systems where file locks are present.

My thinking is that this is a race condition where the file locks cannot be removed before the rm command tries to delete. One potential try is to put a try-except clause in unix.rm that waits a few seconds if it hits this OSError, to give the filesystem some time to remove it's lock.

seisflows/seisflows/tools/unix.py

Lines 197 to 205 in d7661ee

def rm(path):
"""
Remove files or directories
"""
for name in _iterable(path):
if os.path.isfile(name) or os.path.islink(name):
os.remove(name)
elif os.path.isdir(name):
shutil.rmtree(name)

scott314159 commented
bch0w commented

Hey @scott314159, again, sorry for the long pause in communication here! Yeah the log message removal is intended to reduce file count, as for large-scale problems there may be (hundreds of) thousands of files created which may stress file systems, but I agree there is usually useful information there. The parameter export_log_files is meant to make it a User decision to save these files by copying them to a more permanent location, and is set True by default.