NOAA-GFDL/MDTF-diagnostics

associated_files feature causes POD failure

Closed this issue · 0 comments

Bug Severity

  • 1 = Minor problem that does not affect total framework functionality (e.g., computation error in a POD, problem with logging output, or an issue on a single system
  • 2 = Major problem that affects overall functionality, but that does not occur for all users (e.g., problems installing the framework with a specific Conda version, a framework option that causes one or more PODs to fail, or missing/incompatible Python modules).
  • 3 = Catastrophic problem that occurs frequently for multiple users and/or on multiple systems (e.g.,framework consistently fails to install on multiple systems, or one or more PODs continuously fails after running successfully)

Describe the bug
The recent "associated_files" feature causes some PODs to fail in --site NOAA_GFDL model.
The ENSO_RWS POD run on a CM3 dataset with the gfdl naming convention fails with the following error during preprocessing. The issue does not occur with commit 12b91fd

Steps To Reproduce
What steps did you follow to produce the bug
Define following settings in sites/NOAA_GFDL/default_tests.jsonc

case_list: 
        CASENAME: CM3_historical
        model: GFDL-CM3
        convention: GFDL
        FIRSTYR: 2000
        LASTYR: 2004
        CASE_ROOT_DIR: /archive/wnd/MDTF-GFDL/CM3/pp
        pod_list: 
          ENSO_RWS
        experiment: ""
      paths: 
        CODE_ROOT: /home/$USER/mdtf/MDTF-diagnostics
        OBS_DATA_ROOT: /nbhome/jml/mdtf/tmp/inputdata/obs_data
        MODEL_DATA_ROOT: /nbhome/jml/mdtf/tmp/inputdata/model
        WORKING_DIR: /nbhome/jml/mdtf/wkdir
        OUTPUT_DIR: /nbhome/jml/mdtf/mdtf_out
        conda_root: /home/oar.gfdl.mdtf/miniconda3
        conda_env_root: /home/oar.gfdl.mdtf/miniconda3/envs
        OBS_DATA_REMOTE: /home/oar.gfdl.mdtf/mdtf/inputdata/obs_data
        GFDL_PPAN_TEMP: /nbhome/jml/mdtf/tmp
        GFDL_WS_TEMP: /nbhome/jml/mdtf/tmp
        TEMP_DIR_ROOT: /nbhome/jml/mdtf/tmp

Run batch job with contents of mdtf wrapper script modified to point to in-house conda installation on PPAN:

sbatch mdtf_slurm -f sites/NOAA_GFDL/default_gfdl.jsonc --site NOAA_GFDL -v

Environment
Describe the system environment:

  • OS: [e.g. RHEL 7.2] Linux version 3.10.0-1160.62.1.el7.x86_64
  • Conda version: 3.7
  • branch name and link: main branch, tested commits f4cd862, f71adbb

Log information and/or terminal output

12:02:13 DEBUG: log (logs.py line 328):
    Setting experiment_key for 'ts' to '5yr'

12:02:13 DEBUG: log (logs.py line 328):
    DataKey(202) selected as part of experiment_key '('', 'atmos', '5yr')'.

12:02:13 ERROR: log (logs.py line 328):
    Caught exception while setting experiment: AttributeError("'NoneType' object has no attribute 'status'")
    Traceback (most recent call last):
      File "/home/$USER/mdtf/MDTF-diagnostics/src/data_manager.py", line 520, in select_data
        self.set_experiment()
      File "/home/$USER/mdtf/MDTF-diagnostics/src/data_manager.py", line 1190, in set_experiment
        v.associated_files[expt_key].status = core.ObjectStatus.ACTIVE
    AttributeError: 'NoneType' object has no attribute 'status'

12:02:13 ERROR: log (logs.py line 328):
    Caught exception at DataSource level: AttributeError("'NoneType' object has no attribute 'status'").
    Traceback (most recent call last):
      File "/home/$USER/mdtf/MDTF-diagnostics/src/data_manager.py", line 636, in request_data
        self.preprocess_data()
      File "/home/$USER/mdtf/MDTF-diagnostics/src/data_manager.py", line 598, in preprocess_data
        self.fetch_data()
      File "/home/$USER/mdtf/MDTF-diagnostics/src/data_manager.py", line 541, in fetch_data
        self.select_data()
      File "/home/$USER/mdtf/MDTF-diagnostics/src/data_manager.py", line 529, in select_data
        raise exc
      File "/home/$USER/mdtf/MDTF-diagnostics/src/data_manager.py", line 520, in select_data
        self.set_experiment()
      File "/home/$USER/mdtf/MDTF-diagnostics/src/data_manager.py", line 1190, in set_experiment
        v.associated_files[expt_key].status = core.ObjectStatus.ACTIVE
    AttributeError: 'NoneType' object has no attribute 'status'