Unit test: test_restart_serial.py OSError regarding RESTART directory not empty
Closed this issue · 7 comments
Upon a run of the test_restart_serial.py unit test in the tests/main/driver directory, on both the main repo and my fork for fv3core reorganization, the test fails due to an OSError:
FAILED tests/main/driver/test_restart_serial.py::test_restart_save_to_disk - OSError: [Errno 39] Directory not empty: 'RESTART'
The RESTART directory does appear empty during investigation although.
Should we be using the built-in error handling functionality of shutil.rmtree and also invoke a buffer flush by using "import os ; os.sync()" prior to calling shutil.rmtree in finally?
I implemented this and the error still persists unfortunately.
Does it have some invisible files? This has hit me in the past a few times.
It is empty; ls -a only returns the links to the current and parent directories
Interestingly, the test passes in CI
@fmalatino - since you're tests are potentially failing only on gaea, this may be an issue with either the lustre scratch filesystem or the NFS-mounted home filesystem - especially if they are using NFS caching.
Arguably the shutil.rmtree("RESTART") should be shutil.rmtree("RESTART", ignore_errors=True) since this is a utest clean up and shouldn't fail the test since the test is testing something else.
This is fine on my box and on NCCS's Discover (NFS cached filesystem)