model abort due to dvice negative
bingfu-NOAA opened this issue · 11 comments
I was running UFS S2SWA and model crashed with negative dvice error. It is not during the initialization period, the model crashed after the integration into 15-20 days. I was trying to find the discussion page for CICE to post the issue there, but couldn't find it. So I opened this issue here, hopefully get some help.
The CICE hash is from NOAA-EMC/CICE
commit 5840cd1 (HEAD, origin/emc/develop, origin/HEAD, emc/develop)
Merge: 6671e32 7df80ba
Author: Denise Worthen <denise.worthen@noaa.gov>
Date: Wed Mar 22 07:43:35 2023 -0400
The following is the error information.
(shift_ice)shift_ice: negative dvice
(shift_ice)boundary, donor cat: 2 3
(shift_ice)daice = 0.000000000000000E+000
(shift_ice)dvice = -5.200365076407644E-068
(shift_ice)puny = 9.999999999999999E-012
(shift_ice)vicen(nd) = -5.200365076407644E-068
(icepack_warnings_setabort) T :file icepack_itd.F90 :line 549
(shift_ice) shift_ice: negative dvice
(icepack_warnings_aborted) ... (shift_ice)
(icepack_warnings_aborted) ... (linear_itd)
The printout of "puny" and "vicen(nd)" was added by me.
Any of these values make sense?
I attached the code below
if (donor(n) > 0 .and. &
dvice(n) <= -puny*vicen(nd)) then
write(warnstr,*) ' '
call icepack_warnings_add(warnstr)
write(warnstr,*) subname, 'shift_ice: negative dvice'
call icepack_warnings_add(warnstr)
write(warnstr,*) subname, 'boundary, donor cat:', n, nd
call icepack_warnings_add(warnstr)
write(warnstr,*) subname, 'daice =', daice(n)
call icepack_warnings_add(warnstr)
write(warnstr,*) subname, 'dvice =', dvice(n)
call icepack_warnings_add(warnstr)
write(warnstr,*) subname, 'puny =', puny
call icepack_warnings_add(warnstr)
write(warnstr,*) subname, 'vicen(nd) =', vicen(nd)
call icepack_warnings_add(warnstr)
call icepack_warnings_setabort(.true.,__FILE__,__LINE__)
call icepack_warnings_add(subname//' shift_ice: negative dvice')
endif
Thanks,
Bing
I think the question here is how vicen becomes negative? vicen is volume per unit area of ice
It would be good if you can post this on the bulletin board as well.
https://bb.cgd.ucar.edu/cesm/forums/cice-consortium.146/
Now, I do recall there was an issue that was very intermittent where the ITD would produce negative values. It was all about the initialization as I recall. How are you initializing the run here?
@dabail10 Thanks for pointing me to the bulletin board.
I warm start the model with restart files from f03. The model forecast length is set to 35-day, with 300s timestep.
The crash occurred after model integration for 15-20 days, not at the initialization period.
I don't understand what f03 means. Have a look at the netCDF restart file you are using and check the vicen field here.
f03 means restart from forecast hour 03, I checked ice restart file, the vicen value range from 0 to 8.36596.
You should make some plots of aicen and vicen for each category. What is likely happening is that you are initializing with the same thickness in two categories where thickness is vicen / aicen.
@dabail10 more details: We have a continuous coupled run that is nudged to ERA5 in the atmosphere, ORAS5 and the ocean. Sea-ice concentration, thickness, and snow depth are updated once a day at 9Z to the ORAS5 sea-ice using JEDI software.
What Bing uses for his initial conditions are restart files for 3Z the following day, so there has been 18 hours of model integration with no additional input to the sea-ice or ocean, but the atmosphere model continues to be nudged with ERA5.
Bing then launches a free forecast, and the model is crashing in the 3rd week.
Let's move this over to the forums please.
@bingfu-NOAA Can I check if there is any progress on this issue?
I believe Bing is out of tow. The solution we found it to remove small ice fractions from the initial condition file.