associated_files feature causes POD failure
Closed this issue · 0 comments
wrongkindofdoctor commented
Bug Severity
- 1 = Minor problem that does not affect total framework functionality (e.g., computation error in a POD, problem with logging output, or an issue on a single system
- 2 = Major problem that affects overall functionality, but that does not occur for all users (e.g., problems installing the framework with a specific Conda version, a framework option that causes one or more PODs to fail, or missing/incompatible Python modules).
- 3 = Catastrophic problem that occurs frequently for multiple users and/or on multiple systems (e.g.,framework consistently fails to install on multiple systems, or one or more PODs continuously fails after running successfully)
Describe the bug
The recent "associated_files" feature causes some PODs to fail in --site NOAA_GFDL model.
The ENSO_RWS POD run on a CM3 dataset with the gfdl naming convention fails with the following error during preprocessing. The issue does not occur with commit 12b91fd
Steps To Reproduce
What steps did you follow to produce the bug
Define following settings in sites/NOAA_GFDL/default_tests.jsonc
case_list:
CASENAME: CM3_historical
model: GFDL-CM3
convention: GFDL
FIRSTYR: 2000
LASTYR: 2004
CASE_ROOT_DIR: /archive/wnd/MDTF-GFDL/CM3/pp
pod_list:
ENSO_RWS
experiment: ""
paths:
CODE_ROOT: /home/$USER/mdtf/MDTF-diagnostics
OBS_DATA_ROOT: /nbhome/jml/mdtf/tmp/inputdata/obs_data
MODEL_DATA_ROOT: /nbhome/jml/mdtf/tmp/inputdata/model
WORKING_DIR: /nbhome/jml/mdtf/wkdir
OUTPUT_DIR: /nbhome/jml/mdtf/mdtf_out
conda_root: /home/oar.gfdl.mdtf/miniconda3
conda_env_root: /home/oar.gfdl.mdtf/miniconda3/envs
OBS_DATA_REMOTE: /home/oar.gfdl.mdtf/mdtf/inputdata/obs_data
GFDL_PPAN_TEMP: /nbhome/jml/mdtf/tmp
GFDL_WS_TEMP: /nbhome/jml/mdtf/tmp
TEMP_DIR_ROOT: /nbhome/jml/mdtf/tmp
Run batch job with contents of mdtf wrapper script modified to point to in-house conda installation on PPAN:
sbatch mdtf_slurm -f sites/NOAA_GFDL/default_gfdl.jsonc --site NOAA_GFDL -v
Environment
Describe the system environment:
- OS: [e.g. RHEL 7.2] Linux version 3.10.0-1160.62.1.el7.x86_64
- Conda version: 3.7
- branch name and link: main branch, tested commits f4cd862, f71adbb
Log information and/or terminal output
12:02:13 DEBUG: log (logs.py line 328):
Setting experiment_key for 'ts' to '5yr'
12:02:13 DEBUG: log (logs.py line 328):
DataKey(202) selected as part of experiment_key '('', 'atmos', '5yr')'.
12:02:13 ERROR: log (logs.py line 328):
Caught exception while setting experiment: AttributeError("'NoneType' object has no attribute 'status'")
Traceback (most recent call last):
File "/home/$USER/mdtf/MDTF-diagnostics/src/data_manager.py", line 520, in select_data
self.set_experiment()
File "/home/$USER/mdtf/MDTF-diagnostics/src/data_manager.py", line 1190, in set_experiment
v.associated_files[expt_key].status = core.ObjectStatus.ACTIVE
AttributeError: 'NoneType' object has no attribute 'status'
12:02:13 ERROR: log (logs.py line 328):
Caught exception at DataSource level: AttributeError("'NoneType' object has no attribute 'status'").
Traceback (most recent call last):
File "/home/$USER/mdtf/MDTF-diagnostics/src/data_manager.py", line 636, in request_data
self.preprocess_data()
File "/home/$USER/mdtf/MDTF-diagnostics/src/data_manager.py", line 598, in preprocess_data
self.fetch_data()
File "/home/$USER/mdtf/MDTF-diagnostics/src/data_manager.py", line 541, in fetch_data
self.select_data()
File "/home/$USER/mdtf/MDTF-diagnostics/src/data_manager.py", line 529, in select_data
raise exc
File "/home/$USER/mdtf/MDTF-diagnostics/src/data_manager.py", line 520, in select_data
self.set_experiment()
File "/home/$USER/mdtf/MDTF-diagnostics/src/data_manager.py", line 1190, in set_experiment
v.associated_files[expt_key].status = core.ObjectStatus.ACTIVE
AttributeError: 'NoneType' object has no attribute 'status'