Variable attribute 'valid_range'

[Wen-hao-Dong]

Bug Severity

• 1 = Minor problem that does not affect total framework functionality (e.g., computation error in a POD, problem with logging output, or an issue on a single system
• [] 2 = Major problem that affects overall functionality, but that does not occur for all users (e.g., problems installing the framework with a specific Conda version, a framework option that causes one or more PODs to fail, or missing/incompatible Python modules).
• 3 = Catastrophic problem that occurs frequently for multiple users and/or on multiple systems (e.g.,framework consistently fails to install on multiple systems, or one or more PODs continuously fails after running successfully)

Describe the bug
I was applying the ENSO_RWS POD to both CM3 and CM4 historical simulations and found the POD failed for the CM3 outputs. This POD reads in monthly variables, including precip, t_ref, temp, hght, ucomp, vcomp, omega. However, for variables ucomp, vcomp, t_ref, temp in CM3 output, they have an attribute named 'valid_range'. When I ran the POD on these datasets, it failed complaining (take t_ref as an example):

Received event while preprocessing <#hcE2:ENSO_RWS.ts>: DataPreprocessEvent("Caught exception while parsing file metadata for <#hcE2:ENSO_RWS.ts>: ValueError('The truth value of an array with more than one elemen t is ambiguous. Use a.any() or a.all()').")

But removing this attribute will resolve this problem. Can the framework also handle this attribute?

Steps To Reproduce
What steps did you follow to produce the bug

Environment
Describe the system environment:

• OS: [e.g. RHEL 7.2]
• Conda version: [the conda info command will provide the installation information]
• branch name and link: [e.g., develop, https://github.com/NOAA-GFDL/MDTF-diagnostics/tree/develop]

Log information and/or terminal output
/archive/wnd/MDTF-GFDL/STDO/slurm_MDTF_diags.20387294_4294967294.out

[wrongkindofdoctor]

@Wen-hao-Dong I'm having trouble reproducing the error exactly. I found the dataset and output log, but it would help if you could point me to the batch script you submitted, and let me know you if you used additional slurm commands. Also, did you submit this through FREPP, or independtly?

[Wen-hao-Dong]

@wrongkindofdoctor My bad! This is done in the offline mode. Here is the detail information:

1. The MDTF package is installed at: /archive/wnd/MDTF-GFDL/mdtf_0604/MDTF-diagnostics
2. The slurm command I used is: sbatch ./mdtf_slurm -f sites/NOAA_GFDL/default_gfdl.jsonc --site NOAA_GFDL -v
3. The batch job ID is 20888881
4. The STDOUT is at: /archive/wnd/MDTF-GFDL/STDO/slurm_MDTF_diags.20888881_4294967294.out
   Please let me know if you need more information.

[wrongkindofdoctor]

@Wen-hao-Dong Thanks! I didn't realize you were working with an older version of MDTF-diagnostics (commit #12b91fd) which explains why I have encountered a different error during preprocessing

12:58:31 ERROR: log (logs.py line 328):
    Caught exception while setting experiment: AttributeError("'NoneType' object has no attribute 'status'")
    Traceback (most recent call last):
      File "/home/Jessica.Liptak/mdtf/MDTF-diagnostics/src/data_manager.py", line 520, in select_data
        self.set_experiment()
      File "/home/Jessica.Liptak/mdtf/MDTF-diagnostics/src/data_manager.py", line 1190, in set_experiment
        v.associated_files[expt_key].status = core.ObjectStatus.ACTIVE
    AttributeError: 'NoneType' object has no attribute 'status'

12:58:31 ERROR: log (logs.py line 328):
    Caught exception at DataSource level: AttributeError("'NoneType' object has no attribute 'status'").
    Traceback (most recent call last):
      File "/home/Jessica.Liptak/mdtf/MDTF-diagnostics/src/data_manager.py", line 636, in request_data
        self.preprocess_data()
      File "/home/Jessica.Liptak/mdtf/MDTF-diagnostics/src/data_manager.py", line 598, in preprocess_data
        self.fetch_data()
      File "/home/Jessica.Liptak/mdtf/MDTF-diagnostics/src/data_manager.py", line 541, in fetch_data
        self.select_data()
      File "/home/Jessica.Liptak/mdtf/MDTF-diagnostics/src/data_manager.py", line 529, in select_data
        raise exc
      File "/home/Jessica.Liptak/mdtf/MDTF-diagnostics/src/data_manager.py", line 520, in select_data
        self.set_experiment()
      File "/home/Jessica.Liptak/mdtf/MDTF-diagnostics/src/data_manager.py", line 1190, in set_experiment
        v.associated_files[expt_key].status = core.ObjectStatus.ACTIVE
    AttributeError: 'NoneType' object has no attribute 'status'

I'll keep digging.