MOM6 with FMS2 fails when restart's Time axis is not unlimited.
Closed this issue · 9 comments
If MOM is compiled with FMS2 and uses a restart file whose time axis is not unlimited, there is a FMS2 error:
NOTE from PE 0: MOM_restart: MOM run restarted using : INPUT/MOM.res.nc
FATAL from PE 0: NetCDF: Invalid dimension ID or name: get_unlimited_dimension_name: file:INPUT/MOM.res.nc
FATAL from PE 0: NetCDF: Invalid dimension ID or name: get_unlimited_dimension_name: file:INPUT/MOM.res.nc
#0 0x2d6d87e in __mpp_mod_MOD_mpp_error_basic
at MOM6-examples/src/FMS/mpp/include/mpp_util_mpi.inc:72
#1 0x330d3d5 in __fms_io_utils_mod_MOD_error
at MOM6-examples/src/FMS/fms2_io/fms_io_utils.F90:193
#2 0x2234edf in __netcdf_io_mod_MOD_check_netcdf_code
at MOM6-examples/src/FMS/fms2_io/netcdf_io.F90:371
#3 0x22283a3 in __netcdf_io_mod_MOD_get_unlimited_dimension_name
at MOM6-examples/src/FMS/fms2_io/netcdf_io.F90:1316
#4 0xf56cf4 in __mom_io_infra_MOD_get_file_info
at MOM6-examples/src/MOM6/config_src/infra/FMS2/MOM_io_infra.F90:480
#5 0xf568ef in __mom_io_infra_MOD_get_file_times
at MOM6-examples/src/MOM6/config_src/infra/FMS2/MOM_io_infra.F90:488
#6 0x18f72c3 in __mom_io_file_MOD_get_file_times_infra
at MOM6-examples/src/MOM6/src/framework/MOM_io_file.F90:1249
#7 0xe762ae in __mom_restart_MOD_restore_state
at MOM6-examples/src/MOM6/src/framework/MOM_restart.F90:1540
#8 0x1ac1054 in __mom_state_initialization_MOD_mom_initialize_state
at MOM6-examples/src/MOM6/src/initialization/MOM_state_initialization.F90:524
#9 0xba1382 in __mom_MOD_initialize_mom
at MOM6-examples/src/MOM6/src/core/MOM.F90:2834
#10 0x6d5dac in __ocean_model_mod_MOD_ocean_model_init
at MOM6-examples/src/MOM6/config_src/drivers/FMS_cap/ocean_model_MOM.F90:284
#11 0x10adcb5 in coupler_init
at MOM6-examples/src/coupler/coupler_main.F90:1843
#12 0x109d67e in coupler_main
at MOM6-examples/src/coupler/coupler_main.F90:614
#13 0x10afa40 in main
at MOM6-examples/src/coupler/coupler_main.F90:313
The issue seems to be that get_file_info(file, ntimes=...) inside of restore_state assumes that an unlimited dimension exists, and raises an error if it does not.
The FMS1 implementation (mpp_get_times()) reads the time levels from internally stored data, which is compiled when the file is opened, and has a fallback when there is no "recdim" (i.e. unlimited dimension).
I am still unsure to what extent FMS2 handles an absent unlimited, but we will need to provide similar support where it is missing from FMS2.
Thanks to @jiandewang for reporting this.
@jiandewang We've been discussing this. It seems the policy has always been that the time axis should be unlimited and it just happened to work as a fixed dimension with FMS1. It will be quite a bit of work to adapt MOM6 to work for this input when it would be comparatively easy to modify the file with a few lines of python. Thoughts?
@adcroft thanks for the heads up. It's hard for me to give you a black and white answer at this moment. I will need some time to do some testing (should be pretty simple) and thinking about this. My main concern is that these restart files used in UFS have been widely spreaded across UFS community on all sort of platforms (some of them are unknown to me), and I am not the owner for most of them.
Maybe we can still solve this one on our end. @jiandewang @adcroft Do we know if it's safe to assume the time axis is called Time? How about case-insensitive (TIME, time, &c)?
@marshallward it is Time = 1 in our UFS troubled restart file. It is generated from an very old version of MOM6 (at leat 3 or maybe 4 years old). Any new MOM6 restart file generated after that has unlimited in time axis
@jiandewang PR #386 may fix this issue.
@marshallward I am making testing run now, shall be able to tell you the results soon
@marshallward now I only got warning
read_field_3d:time level specified, but the variable u does not have an unlimited dimension in INPUT/MOM.res.nc
job finished OK. thanks
Great, that's also the only change I noticed.
Resolved in #386