National Map Server for 3DEP offline?
Closed this issue · 7 comments
What happened?
I am trying to programmatically download 1 m lidar tiles from the usgs3dep database for a number of sites across the united states. I have written a software package that wraps around by three dab to accomplish this in a randomized way, however the package started failing approximately a month ago. It serves the following error:
Service is currently not available, try again later: https://elevation.nationalmap.gov/arcgis/services/3DEPElevation/ImageServer/WMSServer
What did you expect to happen?
I was expecting to get a tile of lidar data downloaded from the usgs.
Minimal Complete Verifiable Example
import py3dep
bbox = (-78.027678, 39.054019, -77.913653, 39.123205)
dem = py3dep.get_map("DEM", bbox, 1)MVCE confirmation
- Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue.
- Complete example — the example is self-contained, including all data and the text of any traceback.
- New issue — a search of GitHub Issues suggests this is not a duplicate.
Relevant log output
---------------------------------------------------------------------------
CPLE_OpenFailedError Traceback (most recent call last)
File rasterio/_base.pyx:308, in rasterio._base.DatasetBase.__init__()
File rasterio/_base.pyx:219, in rasterio._base.open_dataset()
File rasterio/_err.pyx:221, in rasterio._err.exc_wrap_pointer()
CPLE_OpenFailedError: '/vsimem/8c5edfaa-95e4-4fed-9782-d12c00c6c682/8c5edfaa-95e4-4fed-9782-d12c00c6c682.tif' not recognized as a supported file format.
During handling of the above exception, another exception occurred:
RasterioIOError Traceback (most recent call last)
File ~/mambaforge/envs/levees_usa/lib/python3.11/site-packages/py3dep/py3dep.py:161, in get_map(layers, geometry, resolution, geo_crs, crs)
160 try:
--> 161 ds = geoutils.gtiff2xarray(r_dict, _geometry, crs)
162 except RasterioIOError as ex:
File ~/mambaforge/envs/levees_usa/lib/python3.11/site-packages/pygeoutils/pygeoutils.py:243, in gtiff2xarray(r_dict, geometry, geo_crs, ds_dims, driver, all_touched, nodata, drop)
241 var_name = {lyr: "_".join(lyr.split("_")[:-3]) for lyr in r_dict.keys()}
--> 243 attrs = utils.get_gtiff_attrs(r_dict[key1], ds_dims, driver, nodata)
244 dtypes: dict[str, type] = {}
File ~/mambaforge/envs/levees_usa/lib/python3.11/site-packages/pygeoutils/_utils.py:374, in get_gtiff_attrs(resp, ds_dims, driver, nodata)
373 memfile.write(resp)
--> 374 with memfile.open(driver=driver) as src:
375 r_crs = pyproj.CRS(src.crs)
File ~/mambaforge/envs/levees_usa/lib/python3.11/site-packages/rasterio/env.py:401, in ensure_env.<locals>.wrapper(*args, **kwds)
400 with Env.from_defaults():
--> 401 return f(*args, **kwds)
File ~/mambaforge/envs/levees_usa/lib/python3.11/site-packages/rasterio/io.py:138, in MemoryFile.open(self, driver, width, height, count, crs, transform, dtype, nodata, sharing, **kwargs)
137 log.debug("VSI path: {}".format(mempath.path))
--> 138 return DatasetReader(mempath, driver=driver, sharing=sharing, **kwargs)
139 else:
File rasterio/_base.pyx:310, in rasterio._base.DatasetBase.__init__()
RasterioIOError: '/vsimem/8c5edfaa-95e4-4fed-9782-d12c00c6c682/8c5edfaa-95e4-4fed-9782-d12c00c6c682.tif' not recognized as a supported file format.
The above exception was the direct cause of the following exception:
ServiceUnavailableError Traceback (most recent call last)
Cell In[5], line 2
1 bbox = (-78.027678, 39.054019, -77.913653, 39.123205)
----> 2 dem = py3dep.get_map("DEM", bbox, 1)
File ~/mambaforge/envs/levees_usa/lib/python3.11/site-packages/py3dep/py3dep.py:163, in get_map(layers, geometry, resolution, geo_crs, crs)
161 ds = geoutils.gtiff2xarray(r_dict, _geometry, crs)
162 except RasterioIOError as ex:
--> 163 raise ServiceUnavailableError(wms_url) from ex
164 valid_layers = wms.get_validlayers()
165 return utils.rename_layers(ds, list(valid_layers))
ServiceUnavailableError: Service is currently not available, try again later:
https://elevation.nationalmap.gov/arcgis/services/3DEPElevation/ImageServer/WMSServerAnything else we need to know?
No response
Environment
SYS INFO
commit: None
python: 3.11.1 | packaged by conda-forge | (main, Mar 30 2023, 16:51:07) [GCC 11.3.0]
python-bits: 64
OS: Linux
OS-release: 6.2.0-39-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.12.2
libnetcdf: 4.9.1
PACKAGE VERSION
aiodns N/A
aiohttp 3.8.4
aiohttp-client-cache 0.8.1
aiosqlite 0.18.0
async-retriever 0.15.0
bottleneck 1.3.7
brotli N/A
click 8.1.3
cytoolz 0.12.0
dask 2023.3.2
defusedxml 0.7.1
folium 0.14.0
geopandas 0.12.2
h5netcdf N/A
hydrosignatures N/A
lxml 4.9.2
matplotlib 3.7.1
netCDF4 1.6.3
networkx 3.0
numba N/A
numpy 1.24.2
owslib 0.29.2
pandas 1.5.3
py3dep 0.15.0
pyarrow N/A
pydaymet N/A
pygeohydro N/A
pygeoogc 0.15.0
pygeos N/A
pygeoutils 0.15.0
pynhd N/A
pynldas2 N/A
pyproj 3.5.0
pytest N/A
pytest-cov N/A
rasterio 1.3.6
requests 2.28.2
requests-cache 1.1.0
richdem N/A
rioxarray 0.14.0
scipy 1.10.1
shapely 2.0.1
tables N/A
ujson 5.7.0
urllib3 1.26.15
xarray 2023.3.0
xdist N/A
yaml N/A
Oh I should say that I have already successfully downloaded this particular tile in the past.
Thanks for reporting the issue. It seems that since this is a large request the 3DEP service was being pushed to its limit so it failed. It appears that the 3DEP service capacity is either not as good as before or the number of requests has increased. So, I had to make some modifications to reduce the load of each request. This solved the issue:
I will push this fix. I am planning to release new versions for all HyRiver packages soon. So you can either wait for the official release or install them from git:
pip install git+https://github.com/hyriver/async-retriever.git git+https://github.com/hyriver/pygeoogc.git git+https://github.com/hyriver/pygeoutils.git git+https://github.com/hyriver/py3dep.gitBTW, you can use py3dep.get_dem(bbox, 1) if you only want to get DEM and don't want slope or other layers. This offers the benefit of automatically using the static 3DEP service instead of dynamic for 10, 30, and 60 m resolutions, which can be much faster than the dynamic service. Also, in version 0.16.0 (next release), I added a new function for large requests called py3dep.get_dem_vrt that will create a VRT file instead of loading everything to the memory, so it's much more memory efficient.
Excellent, thanks, I'll work with the new fix in the short term and look for the new stack when you release it.
Our issue is that we only want 1m lidar. Any lower resolution does not satisfy our requirements. So we just discard tiles that lack 1m-resolution coverage in our region of interest.
The new version is already out. Please give it a try and let me know if it works.
I updated py3dep via conda-forge, and it had the same failure. However, I created a fresh environment, with only py3dep as the install, and it works fine. I now suspect there's an issue in my environment preventing the update from working correctly. Instead of trouble-shoot it, I'll just build a fresh environment.
I'm going to close this for now, but if I run into the issue again while re-building the computing environment, I'll let you know where the issue arises.
Thanks for the help!
Great, thanks!