`rmtree` fails due to "directory not empty" on Linux

[christophfriedrich]

Bug report

Bug description:

I have a temp directory in which I have one zip file per day of a month (along the pattern ./agrisens_YYYYMMDD.zip) and the extracted folders (many files along the pattern ./agrisens_YYYYMMDD/foo.bar). The zip files are listed in a downloaded_files variable. My code to clear the temp directory (without deleting it entirely) is like this:

print('Cleaning up temporary files...')
for filename in downloaded_files:
    foldername = filename[0:-4]  # remove last 4 characters (i.e. the file extension (.zip))
    os.remove('temp/'+filename)  # delete zip file
    shutil.rmtree('temp/'+foldername)  # delete folder where zip file was extracted to (os.rmdir only deletes empty directories, but here we still have content in it (and don't want to go through it ourselves))
print('DONE')

Repeatedly, this loop failed half-way through:

Cleaning up temporary files...
Traceback (most recent call last):
  File "/datacube/Scripts/agrisens/dwd_data-v6.py", line 317, in <module>
    shutil.rmtree('temp/'+foldername)  # delete folder where zip file was extracted to (os.rmdir only deletes empty directories, but here we still have content in it (and don't want to go through it ourselves))
  File "/opt/conda/lib/python3.10/shutil.py", line 731, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/opt/conda/lib/python3.10/shutil.py", line 729, in rmtree
    os.rmdir(path)
OSError: [Errno 39] Directory not empty: 'temp/agrisens_20231217'

Interestingly, when looking at the file system, the folder is actually empty!

(base) datacube@d6f03dd71d64:/datacube/Scripts/agrisens/temp$ ll
total 1222793
drwxrws--- 17 datacube 4010225     4096 Dec 18 17:16 ./
drwxrws---  5 datacube 4010225     4096 Dec 18 17:09 ../
drwxrws---  2 datacube 4010225     4096 Dec 18 17:16 agrisens_20231116/
drwxrws---  2 datacube 4010225     4096 Dec 18 16:39 agrisens_20231117/
-rw-rw----  1 datacube 4010225 86510075 Dec 18 16:38 agrisens_20231117.zip
drwxrws---  2 datacube 4010225     4096 Dec 18 16:39 agrisens_20231118/
-rw-rw----  1 datacube 4010225 89981192 Dec 18 16:38 agrisens_20231118.zip
...

(base) datacube@d6f03dd71d64:/datacube/Scripts/agrisens/temp/agrisens_20231217$ ll
total 1
drwxrws---  2 datacube 4010225 4096 Dec 18 18:06 ./
drwxrws--- 17 datacube 4010225 4096 Dec 18 18:06 ../
(base) datacube@d6f03dd71d64:/datacube/Scripts/agrisens/temp/agrisens_20231217$

So it's quite likely that some race condition happened so that the folder was not empty on the attempt to delete it, but shortly after.

Note also that the folder of the day before the criticised folder didn't get removed either. And neither the zip file of the day in question, despite the call to os.remove being before the call to shutil.rmtree in the loop.

---

The same issue also happened with the previous month:

Cleaning up temporary files...
Traceback (most recent call last):
  File "/datacube/Scripts/agrisens/dwd_data-v6.py", line 317, in <module>
    shutil.rmtree('temp/'+foldername)  # delete folder where zip file was extracted to (os.rmdir only deletes empty directories, but here we still have content in it (and don't want to go through it ourselves))
  File "/opt/conda/lib/python3.10/shutil.py", line 731, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/opt/conda/lib/python3.10/shutil.py", line 729, in rmtree
    os.rmdir(path)
OSError: [Errno 39] Directory not empty: 'temp/agrisens_20231116'

In both cases, the fail happened quite exactly in the middle of the month -- might be coincidental, might be noteworthy.

---

Has this been discussed elsewhere?

I looked through open issues with "rmtree" and saw two that might be related:

• The bug sounds similar to #84324, but for me it's happening on Linux, not on Windows
• The actual bug might be workaroundable through #52769

Other notes
The file system being written to is a network storage that is mounted into a virtual machine, from where it is mounted into the Docker container where my actual code runs, so not the closest setup -> a good environment for race conditions to happen. To my knowledge, no Windows is involved anywhere in the chain.

Version

> python3 --version
Python 3.10.12

CPython versions tested on:

3.10

Operating systems tested on:

Linux