test seaice_suite POD with ESM4 failed

Question

test seaice_suite POD with ESM4 failed

Wen-hao-Dong opened this issue 2 years ago · 4 comments

Bug Severity

1 = Minor problem that does not affect total framework functionality (e.g., computation error in a POD, problem with logging output, or an issue on a single system
2 = Major problem that affects overall functionality, but that does not occur for all users (e.g., problems installing the framework with a specific Conda version, a framework option that causes one or more PODs to fail, or missing/incompatible Python modules).
3 = Catastrophic problem that occurs frequently for multiple users and/or on multiple systems (e.g.,framework consistently fails to install on multiple systems, or one or more PODs continuously fails after running successfully)

Describe the bug
Data request for <#N4W2:seaice_suite.siconc> completed succesfully. And the netcdf data has been saved correctly (/archive/wnd/MDTF-GFDL/wkdir/MDTF_ESM4_historical_D1_2000_2004.v3/seaice_suite/model). However, the POD failed to be executed. So I assume this may be related to part of the POD script. One thing I want to mention here is the package choose the component to read in the siconc, which is on the original ocean T grid. I do not know what would be the results if it read in the siconc from the ice_1x1 component, which on the standard lat-lon grid.

Start execution of <#N0r0:ESM4_historical_D1.seaice_suite>

_CONDA_EXE=/home/oar.gfdl.mdtf/miniconda3/bin/conda
_CONDA_ROOT=/home/oar.gfdl.mdtf/miniconda3
Found program python3.
/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/numpy/lib/nanfunctions.py:1665: RuntimeWarning: Degrees of freedom <= 0 for slice.
keepdims=keepdims)
/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/numpy/lib/nanfunctions.py:1665: RuntimeWarning: Degrees of freedom <= 0 for slice.
keepdims=keepdims)
/arch0/wnd/MDTF-GFDL/mdtf/MDTF-diagnostics/diagnostics/seaice_suite/seaice_MLD_stats.py:76: RuntimeWarning: invalid value encountered in true_divide
r = r_num / r_den
/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/numpy/lib/nanfunctions.py:1665: RuntimeWarning: Degrees of freedom <= 0 for slice.
keepdims=keepdims)
/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/numpy/lib/nanfunctions.py:1665: RuntimeWarning: Degrees of freedom <= 0 for slice.
keepdims=keepdims)
/arch0/wnd/MDTF-GFDL/mdtf/MDTF-diagnostics/diagnostics/seaice_suite/seaice_MLD_stats.py:76: RuntimeWarning: invalid value encountered in true_divide
r = r_num / r_den
/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/cartopy/crs.py:228: ShapelyDeprecationWarning: len for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the geoms property instead to get the number of parts of a multi-part geometry.
if len(multi_line_string) > 1:
/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/cartopy/crs.py:280: ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the geoms property to access the constituent parts of a multi-part geometry.
for line in multi_line_string:
/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/cartopy/crs.py:347: ShapelyDeprecationWarning: len for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the geoms property instead to get the number of parts of a multi-part geometry.
if len(p_mline) > 0:
/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/cartopy/crs.py:385: ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the geoms property to access the constituent parts of a multi-part geometry.
line_strings.extend(multi_line_string)
/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/cartopy/crs.py:385: ShapelyDeprecationWarning: len for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the geoms property instead to get the number of parts of a multi-part geometry.
line_strings.extend(multi_line_string)
/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/cartopy/io/init.py:241: DownloadWarning: Downloading: https://naturalearth.s3.amazonaws.com/110m_physical/ne_110m_land.zip
warnings.warn('Downloading: {}'.format(url), DownloadWarning)
file /vftmp/Wenhao.Dong/job24888891/wkdir/MDTF_ESM4_historical_D1_2000_2004.v3/seaice_suite/mon/ESM4_historical_D1.siconc.mon.nc
Limit domain to Arctic to match obs
varname siconc
file /vftmp/Wenhao.Dong/job24888891/inputdata/obs_data/seaice_suite/HadISST_ice_1979-2016_grid_nh.nc
Limit domain to Arctic to match obs
varname sic
computing trend, this may take a few minutes
computing the rest
main stats done
computing trend, this may take a few minutes
computing the rest
main stats done
Traceback (most recent call last):
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/urllib/request.py", line 1350, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/http/client.py", line 1281, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/http/client.py", line 1327, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/http/client.py", line 1276, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/http/client.py", line 1036, in _send_output
self.send(msg)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/http/client.py", line 976, in send
self.connect()
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/http/client.py", line 1443, in connect
super().connect()
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/http/client.py", line 948, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/socket.py", line 728, in create_connection
raise err
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/arch0/wnd/MDTF-GFDL/mdtf/MDTF-diagnostics/diagnostics/seaice_suite/seaice_suite_sic_mean_sigma.py", line 311, in
statname='Mean', unitname='Fraction', vmin=0., vmax =1.)
File "/arch0/wnd/MDTF-GFDL/mdtf/MDTF-diagnostics/diagnostics/seaice_suite/seaice_suite_sic_mean_sigma.py", line 303, in monthlyplot
plt.savefig(figfile, format='png',dpi=300)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/matplotlib/pyplot.py", line 859, in savefig
res = fig.savefig(*args, kwargs)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/matplotlib/figure.py", line 2311, in savefig
self.canvas.print_figure(fname, kwargs)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/matplotlib/backend_bases.py", line 2217, in print_figure
**kwargs)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/matplotlib/backend_bases.py", line 1639, in wrapper
return func(*args, **kwargs)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/matplotlib/backends/backend_agg.py", line 509, in print_png
FigureCanvasAgg.draw(self)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/matplotlib/backends/backend_agg.py", line 407, in draw
self.figure.draw(self.renderer)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/matplotlib/artist.py", line 41, in draw_wrapper
return draw(artist, renderer, *args, **kwargs)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/matplotlib/figure.py", line 1864, in draw
renderer, self, artists, self.suppressComposite)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/matplotlib/image.py", line 131, in _draw_list_compositing_images
a.draw(renderer)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/matplotlib/artist.py", line 41, in draw_wrapper
return draw(artist, renderer, args, kwargs)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/cartopy/mpl/geoaxes.py", line 517, in draw
return matplotlib.axes.Axes.draw(self, renderer=renderer, kwargs)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/matplotlib/artist.py", line 41, in draw_wrapper
return draw(artist, renderer, args, **kwargs)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/matplotlib/cbook/deprecation.py", line 411, in wrapper
return func(*inner_args, **inner_kwargs)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/matplotlib/axes/_base.py", line 2747, in draw
mimage._draw_list_compositing_images(renderer, self, artists)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/matplotlib/image.py", line 131, in _draw_list_compositing_images
a.draw(renderer)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/matplotlib/artist.py", line 41, in draw_wrapper
return draw(artist, renderer, *args, kwargs)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/cartopy/mpl/feature_artist.py", line 153, in draw
geoms = self._feature.intersecting_geometries(extent)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/cartopy/feature/init.py", line 297, in intersecting_geometries
return super().intersecting_geometries(extent)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/cartopy/feature/init.py", line 106, in intersecting_geometries
return (geom for geom in self.geometries() if
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/cartopy/feature/init.py", line 281, in geometries
name=self.name)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/cartopy/io/shapereader.py", line 282, in natural_earth
return ne_downloader.path(format_dict)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/cartopy/io/init.py", line 203, in path
result_path = self.acquire_resource(target_path, format_dict)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/cartopy/io/shapereader.py", line 337, in acquire_resource
shapefile_online = self._urlopen(url)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/site-packages/cartopy/io/init**.py", line 242, in _urlopen
return urlopen(url)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/urllib/request.py", line 1393, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/home/oar.gfdl.mdtf/miniconda3/envs/_MDTF_python3_base/lib/python3.7/urllib/request.py", line 1352, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 110] Connection timed out>

<#N0r0:ESM4_historical_D1.seaice_suite> was terminated or exited abnormally.
Log for <#N0r0:ESM4_historical_D1.seaice_suite>:
*** caught exception (#1):
None: None

INFO: 	Running seaice_suite_sic_mean_sigma.py for <#N0r0:ESM4_historical_D1.seaice_suite>.
INFO: <#N0r0:ESM4_historical_D1.seaice_suite> was terminated or exited abnormally.
INFO: Checking linked output files for <#N0r0:ESM4_historical_D1.seaice_suite>.
ERROR: Missing '$WK_DIR/seaice_suite/model/themean_2000-2004.png'.
ERROR: Missing '$WK_DIR/seaice_suite/model/themean_anomalies_2000-2004.png'.
ERROR: Missing '$WK_DIR/seaice_suite/model/trend_2000-2004.png'.
ERROR: Missing '$WK_DIR/seaice_suite/model/trend_anomalies_2000-2004.png'.
ERROR: Missing '$WK_DIR/seaice_suite/model/detrendedstd_2000-2004.png'.
ERROR: Missing '$WK_DIR/seaice_suite/model/detrendedstd_anomalies_2000-2004.png'.
ERROR: Missing '$WK_DIR/seaice_suite/model/onemolagcorr_2000-2004.png'.
ERROR: Missing '$WK_DIR/seaice_suite/model/onemolagcorr_anomalies_2000-2004.png'.
ERROR: Missing '$WK_DIR/seaice_suite/model/oneyrlagcorr_2000-2004.png'.
ERROR: Missing '$WK_DIR/seaice_suite/model/oneyrlagcorr_anomalies_2000-2004.png'.
ERROR: Deactivated <#N0r0:ESM4_historical_D1.seaice_suite> due to MDTFFileNotFoundError("[Errno 2] No such file or directory: 'Missing 10 files.'").

Steps To Reproduce
I am using the updated version and have submitted the job using the following command: sbatch ./mdtf_slurm -f sites/NOAA_GFDL/default_gfdl.jsonc --site NOAA_GFDL -v
Here is the path to my test repo: /archive/wnd/MDTF-GFDL/mdtf/MDTF-diagnostics
The branch name and link are: test0716, https://github.com/Wen-hao-Dong/MDTF-diagnostics/tree/test0716

Environment
Describe the system environment:

OS: [e.g. RHEL 7.2]
Conda version: [the conda info command will provide the installation information]
branch name and link: [e.g., test0716, https://github.com/Wen-hao-Dong/MDTF-diagnostics/tree/test0716]

Log information and/or terminal output
/archive/wnd/MDTF-GFDL/wkdir/MDTF_ESM4_historical_D1_2000_2004.v3/seaice_suite/seaice_suite.log

since i noticed the time range in the code is hard coded (set to 1979 to 2014), I also test if we use the same range would solve the problem, but turns out it has the same issue:
/archive/wnd/MDTF-GFDL/wkdir/MDTF_ESM4_historical_D1_1979_2014/seaice_suite/seaice_suite.log

Answer 1 · 2022-09-22T19:57:19.000Z

@Wen-hao-Dong This one looks like a cartopy issue. I'll try updating the version in the python3 environment file, and will let you know when the changes are merged into the gfdl MDTF-diagnostics installation

Answer 2 · 2022-09-23T09:45:43.000Z

@Wen-hao-Dong I updated cartopy to v 20, which should solve the https issue. The GFDL python3_base env has the update, so retest when you are ready.

Answer 3 · 2022-09-23T17:29:32.000Z

@wrongkindofdoctor I updated my repo and rerun the test, but still got the same problem.
Please see the log file here: /archive/wnd/MDTF-GFDL/wkdir/MDTF_ESM4_historical_D1_2000_2004.v5/seaice_suite/seaice_suite.log

Answer 4 · 2022-09-23T17:39:35.000Z

@Wen-hao-Dong Okay, thanks for checking. I'm going to put this issue on hiatus for the next week or so I don't get behind on the multirun stuff again. I'm also trying to put in a fix for the albedofb POD. I'll get done what I can today, and push everything to a test branch in my repo for reference. I will note this in #404.