problems reading CESM files
Closed this issue · 8 comments
- [2] 2 = Major problem that affects overall functionality, but that does not occur for all users
Describe the bug
The data_manager seems to be successfully reading the CESM files and writing its own version, but the framework code seems to fail when trying to read in the newly written file (as best as I can tell). I'd just like to go through with someone to try to figure out what is not working in the data_manager (or beyond).
Steps To Reproduce
Standard MDTF run with Wheeler-Kiladis pod. I can share the data and run files.
Environment
Describe the system environment:
- OS: SUSE Linux (NCAR's Cheyenne machine)
- Conda version: 4.12.0
- branch name and link:
GFDL Main bitterbark@9b9f68d
Log information and/or terminal output
bhist.e213cosp217.f09_g17.mdtf_v3.1993.data.log
out.txt
Hi, @bitterbark. It looks like there is an attribute in your data that the framework is having trouble processing before writing the new output files based on this error:
TypeError('cannot perform reduce with flexible type').")
The framework has been encountering more of these kinds of errors lately as users test in on new datasets with attributes that are not single string values that the framework assumes, so this probably isn't an issue I can walk you through until I have identified the cause.
If you can send me the data--either ftp link or globus (share with jessica.liptak@rdhpcs.noaa.gov)--I can run it through the debugger and see where the sticking point(s) is(are).
Thanks @wrongkindofdoctor . I've posted the input files on ftp://ftp.cgd.ucar.edu/archive/mdtf/timeslice_test
(log in as guest). Let me know if there's something I can do on my end.
@bitterbark The directory is missing the variable OMEGA500 required for the W-K POD
OMEGA is commented out in the requirements, see attached. Also this file has the mod necessary to use RLNT instead of RLUT, my apologies for not including it before
settings.jsonc.txt.
Even if the OMEGA or RLUT is not working, the framework rejects all of the files:
DEBUG: Data request for <#5PwD:Wheeler_Kiladis.rlnt> failed.
DEBUG: Data request for <#5Pzw:Wheeler_Kiladis.pr> failed.
DEBUG: Data request for <#5PCy:Wheeler_Kiladis.u200> failed.
DEBUG: Data request for <#5QjC:Wheeler_Kiladis.u200> failed.
DEBUG: Data request for <#5PFx:Wheeler_Kiladis.u850> failed.
DEBUG: Data request for <#5Qtw:Wheeler_Kiladis.u850> failed.
However, I just got the framework to accept the data by stripping out all extra fields in the files.
DEBUG: Data request for <#gEod:Wheeler_Kiladis.rlnt> completed succesfully.
DEBUG: Data request for <#gEsA:Wheeler_Kiladis.pr> completed succesfully.
DEBUG: Data request for <#gEwU:Wheeler_Kiladis.u200> completed succesfully.
DEBUG: Data request for <#gFfX:Wheeler_Kiladis.u200> not used.
DEBUG: Data request for <#gEzY:Wheeler_Kiladis.u850> completed succesfully.
DEBUG: Data request for <#gFoL:Wheeler_Kiladis.u850> not used.
DEBUG: Data request for <#gEgK:bhist.e213cosp217.f09_g17.mdtf_v3.1993.Wheeler_Kiladis> completed succesfully.
I didn't see any attributes that weren't single strings but clearing out unnecessary fields is a step I can add to the processing I do to make the timeseries files. It will be an extra ask for other CESM users, though. I wonder if the data manager could simply look for what it needs in the file and ignore any extras?
Also I'm not sure why it has repeat instances for u200 & u850 (in the quoted output above).
(Note that POD figures aren't yet being produced I just didn't want you to have to repeat this step).
@bitterbark Thank you! I added FLNT to the NCAR fieldlist, and was able to replicate the original error using the modified settings.jsonc file. I've narrowed the problem attributes to the _Fillvalues = '-' defined for the character variables date_written and time_written. I think this is because the framework expects the FillValue to be numeric, not a string, but I'll have to do some more digging.
@bitterbark It turns out the the problem attributes are type bytes, not string, but I have put in a fix for the preprocessor to handle attributes with this type that will get your POD to run.
Subsequent errors seem to be NCL-related. I don't know about the others, but the OMEGA_500 error is because you need to ignore the file environment variables omega500_var and OMEGA500_FILE, since they are present in the settings.jsonc file for this case:
COMPUTING THE SPACE-TIME SPECTRA
starting var u200_var
NCL routine /home/jessica.liptak/mdtf/MDTF-diagnostics/diagnostics/Wheeler_Kiladis/wkSpaceTime_driver.ncl
Copyright (C) 1995-2019 - All Rights Reserved
University Corporation for Atmospheric Research
NCAR Command Language Version 6.6.2
The use of this software is governed by a License Agreement.
See http://www.ncl.ucar.edu/ for more details.
warning:FileSetFileOption: invalid value supplied for option Format
warning:FileSetFileOption: invalid value supplied for option Format
warning:FileSetFileOption: invalid value supplied for option Format
warning:FileSetFileOption: invalid value supplied for option Format
warning:FileSetFileOption: invalid value supplied for option Format
warning:FileSetFileOption: invalid value supplied for option Format
warning:FileSetFileOption: invalid value supplied for option Format
(0) varName:u200
(0) diri:/home/jessica.liptak/mdtf/wkdir/MDTF_bhist.e213cosp217.f09_g17.mdtf_v3.1993_1993_1994/Wheeler_Kiladis/day/
(0) case:bhist.e213cosp217.f09_g17.mdtf_v3.1993
(0) writing to /home/jessica.liptak/mdtf/wkdir/MDTF_bhist.e213cosp217.f09_g17.mdtf_v3.1993_1993_1994/Wheeler_Kiladis/model/PS/
fatal:The result of the conditional expression yields a missing value. NCL can not determine branch, see ismissing function
fatal:["Execute.c":8637]:Execute: Error occurred at or near line 154 in file /home/jessica.liptak/mdtf/MDTF-diagnostics/diagnostics/Wheeler_Kiladis/wkSpaceTime_driver.ncl
starting var u850_var
NCL routine /home/jessica.liptak/mdtf/MDTF-diagnostics/diagnostics/Wheeler_Kiladis/wkSpaceTime_driver.ncl
Copyright (C) 1995-2019 - All Rights Reserved
University Corporation for Atmospheric Research
NCAR Command Language Version 6.6.2
The use of this software is governed by a License Agreement.
See http://www.ncl.ucar.edu/ for more details.
warning:FileSetFileOption: invalid value supplied for option Format
warning:FileSetFileOption: invalid value supplied for option Format
warning:FileSetFileOption: invalid value supplied for option Format
warning:FileSetFileOption: invalid value supplied for option Format
warning:FileSetFileOption: invalid value supplied for option Format
warning:FileSetFileOption: invalid value supplied for option Format
warning:FileSetFileOption: invalid value supplied for option Format
(0) varName:u850
(0) diri:/home/jessica.liptak/mdtf/wkdir/MDTF_bhist.e213cosp217.f09_g17.mdtf_v3.1993_1993_1994/Wheeler_Kiladis/day/
(0) case:bhist.e213cosp217.f09_g17.mdtf_v3.1993
(0) writing to /home/jessica.liptak/mdtf/wkdir/MDTF_bhist.e213cosp217.f09_g17.mdtf_v3.1993_1993_1994/Wheeler_Kiladis/model/PS/
fatal:The result of the conditional expression yields a missing value. NCL can not determine branch, see ismissing function
fatal:["Execute.c":8637]:Execute: Error occurred at or near line 154 in file /home/jessica.liptak/mdtf/MDTF-diagnostics/diagnostics/Wheeler_Kiladis/wkSpaceTime_driver.ncl
starting var omega500_var
Traceback (most recent call last):
File "/home/jessica.liptak/mdtf/MDTF-diagnostics/diagnostics/Wheeler_Kiladis/Wheeler_Kiladis.py", line 58, in <module>
if os.path.isfile(os.environ[file_]):
File "/home/jessica.liptak/miniconda3/envs/_MDTF_NCL_base/lib/python3.7/os.py", line 681, in __getitem__
raise KeyError(key) from None
KeyError: 'OMEGA500_FILE'
--------------------------------------------------------------------------------
<#4ssZ:bhist.e213cosp217.f09_g17.mdtf_v3.1993.Wheeler_Kiladis> was terminated or exited abnormally.
Log for <#4ssZ:bhist.e213cosp217.f09_g17.mdtf_v3.1993.Wheeler_Kiladis>:
*** caught exception (#1):
None: None