Unable to load DEM data with get_map
colby-thrash opened this issue · 8 comments
What happened?
I followed one of the examples to get a basin geometry and download corresponding DEM data. I am able to get the DEM data with .get_dem but get an error using .get_map.
I am new to hyriver and was able to follow many of the examples the first day I downloaded and experimented with no issues. A few days later I came back to the same scripts, and they were no longer working as before. With some troubleshooting/experimentation I discovered that .get_dem worked while I continued to get an error with .get_map.
What did you expect to happen?
I expected both methods of downloading DEM to work
Minimal Complete Verifiable Example
import py3dep
from pynhd import NLDI
geom = NLDI().get_basins("01031500").geometry.iloc[0]
dem1 = py3dep.get_dem(geom, resolution=30)
dem2 = py3dep.get_map("DEM", geom, resolution=30, geo_crs="epsg:4326", crs="epsg:3857")MVCE confirmation
- Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue.
- Complete example — the example is self-contained, including all data and the text of any traceback.
- New issue — a search of GitHub Issues suggests this is not a duplicate.
Relevant log output
runcell(1, 'C:/Users/nrthrac/Desktop/working/MWSS/python-gis/pyhydro-load-dem.py')
Traceback (most recent call last):
File ~\.conda\envs\hyriver\lib\site-packages\urllib3\connectionpool.py:715 in urlopen
httplib_response = self._make_request(
File ~\.conda\envs\hyriver\lib\site-packages\urllib3\connectionpool.py:404 in _make_request
self._validate_conn(conn)
File ~\.conda\envs\hyriver\lib\site-packages\urllib3\connectionpool.py:1058 in _validate_conn
conn.connect()
File ~\.conda\envs\hyriver\lib\site-packages\urllib3\connection.py:419 in connect
self.sock = ssl_wrap_socket(
File ~\.conda\envs\hyriver\lib\site-packages\urllib3\util\ssl_.py:449 in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(
File ~\.conda\envs\hyriver\lib\site-packages\urllib3\util\ssl_.py:493 in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File ~\.conda\envs\hyriver\lib\ssl.py:513 in wrap_socket
return self.sslsocket_class._create(
File ~\.conda\envs\hyriver\lib\ssl.py:1104 in _create
self.do_handshake()
File ~\.conda\envs\hyriver\lib\ssl.py:1375 in do_handshake
self._sslobj.do_handshake()
SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File ~\.conda\envs\hyriver\lib\site-packages\requests\adapters.py:486 in send
resp = conn.urlopen(
File ~\.conda\envs\hyriver\lib\site-packages\urllib3\connectionpool.py:799 in urlopen
retries = retries.increment(
File ~\.conda\envs\hyriver\lib\site-packages\urllib3\util\retry.py:592 in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
MaxRetryError: HTTPSConnectionPool(host='elevation.nationalmap.gov', port=443): Max retries exceeded with url: /arcgis/services/3DEPElevation/ImageServer/WMSServer?service=WMS&request=GetCapabilities&version=1.3.0 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File ~\.conda\envs\hyriver\lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec
exec(code, globals, locals)
File c:\users\nrthrac\desktop\working\mwss\python-gis\pyhydro-load-dem.py:42
dem2 = py3dep.get_map("DEM", geom, resolution=30, geo_crs="epsg:4326", crs="epsg:3857")
File ~\.conda\envs\hyriver\lib\site-packages\py3dep\py3dep.py:164 in get_map
valid_layers = wms.get_validlayers()
File ~\.conda\envs\hyriver\lib\site-packages\pygeoogc\pygeoogc.py:348 in get_validlayers
return self.client.get_validlayers()
File ~\.conda\envs\hyriver\lib\site-packages\pygeoogc\core.py:456 in get_validlayers
wms = WebMapService(self.url, version=self.version)
File ~\.conda\envs\hyriver\lib\site-packages\owslib\wms.py:54 in WebMapService
return wms130.WebMapService_1_3_0(
File ~\.conda\envs\hyriver\lib\site-packages\owslib\map\wms130.py:75 in __init__
self._capabilities = reader.read(self.url, timeout=self.timeout)
File ~\.conda\envs\hyriver\lib\site-packages\owslib\map\common.py:65 in read
u = openURL(spliturl[0], spliturl[1], method='Get',
File ~\.conda\envs\hyriver\lib\site-packages\owslib\util.py:209 in openURL
req = requests.request(method.upper(), url_base, headers=headers, **rkwargs)
File ~\.conda\envs\hyriver\lib\site-packages\requests\api.py:59 in request
return session.request(method=method, url=url, **kwargs)
File ~\.conda\envs\hyriver\lib\site-packages\requests\sessions.py:589 in request
resp = self.send(prep, **send_kwargs)
File ~\.conda\envs\hyriver\lib\site-packages\requests\sessions.py:703 in send
r = adapter.send(request, **kwargs)
File ~\.conda\envs\hyriver\lib\site-packages\requests\adapters.py:517 in send
raise SSLError(e, request=request)
SSLError: HTTPSConnectionPool(host='elevation.nationalmap.gov', port=443): Max retries exceeded with url: /arcgis/services/3DEPElevation/ImageServer/WMSServer?service=WMS&request=GetCapabilities&version=1.3.0 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)')))Anything else we need to know?
No response
Environment
PACKAGE VERSION
aiohttp 3.8.5
aiohttp-client-cache 0.9.1
aiosqlite 0.18.0
async-retriever 0.15.2
bottleneck 1.3.5
click 8.1.7
cytoolz 0.12.0
defusedxml 0.7.1
folium 0.14.0
geopandas 0.14.0
h5netcdf 1.2.0
hydrosignatures 0.15.2
joblib 1.2.0
matplotlib 3.8.0
multidict 6.0.2
netcdf4 1.6.4
networkx 3.1
numba N/A
numpy 1.26.0
owslib 0.29.2
pandas 2.1.1
py3dep 0.15.2
py7zr N/A
pyarrow 13.0.0
pydaymet 0.15.2
pyflwdir N/A
pygeohydro 0.15.2
pygeoogc 0.15.2
pygeoutils 0.15.2
pynhd 0.15.2
pynldas2 0.15.2
pyogrio N/A
pyproj 3.6.1
rasterio 1.3.9
requests 2.31.0
requests-cache 1.1.0
rioxarray 0.15.0
scipy 1.11.3
shapely 2.0.2
ujson 5.4.0
url-normalize 1.4.3
urllib3 1.26.18
xarray 2023.6.0
yarl 1.8.1
It was probably due to on internal issue with the 3DEP web service. I just tested it, and it works without any issue. Please try again and let me know if it works.
I tried running again today and get the same error for dem2.
I am also getting a new error for dem1... GEOSException: Shell empty after removing invalid points I am using Spyder IDE. If I run the same script in my conda prompt with the same environment I do not get the error for dem1 but I still get the dem2 error. I think the problem is most likely on my end. I'm running on my work network which is pretty restricted and apparently I'm having an issue running it in Spyder. I have very limited knowledge of web services etc (which is why HyRiver is an excellent tool for me), but I'm confused at why it worked great initially but is no longer. I can create a new environment if the problem could be there but I'm not sure what else to try. I'm hesitant to build my workflow around something I've only had intermittent success with. I don't think the problem is on your end but any advice would be appreciated.
It can be related to an issue with your network, since I just ran your code and both worked.
What's your use case?
The get_dem function uses an additional data source, compared to get_map. get_dem can be much faster than get_map for these three resolutions, since get_map uses the dynamic layer of 3DEP that tends to be slower. But get_map gives you many more options and flexibility in terms of application. get_dem on the other hand, uses the static maps for 10, 30, and 60 m resolutions and for any other resolution falls back to using get_map under the hood. So, I'd recommend using get_dem if you only want DEM data. If you want other topography data such as slope and hillshade you should use get_map.
To pinpoint the issue, please create a new env with only pynhd and py3dep installed and try again. Assuming you're using mamba for env management, do this:
mamba create -n dem py3dep pynhd ipython
Then activate this new env and run ipython and copy and paste your code there.
Mine worked:
I tried again with a fresh environment and get the same error. I also tried the same setup on another machine and get the same error.
Ultimately my use case is to create HEC-HMS models for multiple watersheds. I wanted to use HyRiver tools to clip my DEMs to the watersheds of interest and also pull in river and soils data without using manual downloads and arcgis. It seems I have a work around for DEMs but I encountered the same error elsewhere in the HyRiver tool chain. I used DEM for a succinct example hoping to solve the problem.
It must be a network issue. I switched to our public wifi and ran it several times with no issues. Thanks for the responses and for the great tool.
I actually created such a workflow for generating input data for a HEC-HMS model when I was TA of a hydrology course. Glad you figured out the issue. It's probably due to SSL certification setup on the network that you were using. I might have a workaround for that, so if you want to use it on that network, please let me know, and I will make some modifications to see if it solves your issue.
I don't think there is any need to create a workaround for my use.
Since you have experience with this specific application, I have another question if you are willing to answer. Once I download and save the DEM as a .tif using py3dep and bring it into HEC-HMS I am having trouble with the HMS GIS tools. I attached two screen shots of my flow accumulation grids obtained from the Preprocess Drainage tool. It seems that HMS is using values at the extent of the clipped DEM? The DEM terrain displays only the clipped watershed in HMS as well as in ArcGIS. The range of the DEM values seem reasonable and do not contain 0s. I tried different projections, adjusting parameters of to_raster, etc., and have gotten the same results with two different watersheds. I am using HMS 4.11 and will include example code used to get the DEM. I'm not sure where the problem lies. I have used clipped DEMs obtained with ArcGIS successfully in HMS. Am I doing something wrong here?
import py3dep
from pynhd import NLDI
basin = NLDI().get_basins('06896189') #stanberry
geom = basin.geometry.iloc[0]
dem = py3dep.get_dem(geom, resolution=30)
dem.rio.to_raster('dem-stanberry.tif')
You should get the DEM for the bound of the polygon and even add a bit of buffer. You can then clip your DEM in HEC-HMS during the basin delineation process.
In your example, since you're getting the data at 30-m res, you can add a buffer of 100 m. Usually 3 to 5 times the resolution is good enough.
import py3dep
from pynhd import NLDI
basin = NLDI().get_basins('06896189') #stanberry
geom = basin.to_crs(5070).buffer(100).to_crs(4326).geometry.iloc[0].bounds
dem = py3dep.get_dem(geom, resolution=30)
dem.rio.to_raster('dem-stanberry.tif')