Failing to pull non-static DEM

[gjkoplik]

What happened?

My use of py3dep that had been working until this past week (as tested by CI) is now failing to pull 3DEP data at a non-static resolution. Note: this does not appear to relate any releases on your end, but the error appears to go from py3dep into pygeoogc into async-retriever, where I'm guessing the issue is stemming from my underlying aiohttp-client-cache going from 0.8.2 to 0.9.1. Also note I am installing py3dep with pip not conda (reproducible example below).

What did you expect to happen?

Pull the DEM data into memory as an xr.Dataset.

Minimal Complete Verifiable Example

Conda env yml file:

name: py3dep_env
channels:
- conda-forge
dependencies:
    - python=3.11
    - pip
    - pip:
      - py3dep

Python stress test that should succeed on static and fail on non-static request:

import py3dep
from shapely.geometry import box

xmin = -120.8
ymin = 39.0
xmax = -120.5
ymax = 39.3

bounds = box(xmin, ymin, xmax, ymax)
bounds_crs = 4326

# static dems should work
ds = py3dep.get_dem(geometry=bounds, resolution=30, crs=bounds_crs)
print("static dem success")

# calculated dem should fail
ds = py3dep.get_dem(geometry=bounds, resolution=120, crs=bounds_crs)
print("calculated dem success")

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

Traceback (most recent call last):
  File "/home/gary.koplik/repos/geosnipe/sandbox/stress_test_py3dep.py", line 22, in <module>
    ds = py3dep.get_dem(geometry=bounds, resolution=120, crs=bounds_crs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gary.koplik/.conda/envs/py3dep_env/lib/python3.11/site-packages/py3dep/py3dep.py", line 298, in get_dem
    dem = get_map("DEM", geometry, resolution, crs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gary.koplik/.conda/envs/py3dep_env/lib/python3.11/site-packages/py3dep/py3dep.py", line 148, in get_map
    valid_crs = ogc_utils.valid_wms_crs(wms_url)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gary.koplik/.conda/envs/py3dep_env/lib/python3.11/site-packages/pygeoogc/utils.py", line 796, in valid_wms_crs
    root = ETree.fromstring(ar.retrieve_text([url], [kwds], ssl=False)[0])
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gary.koplik/.conda/envs/py3dep_env/lib/python3.11/site-packages/async_retriever/async_retriever.py", line 498, in retrieve_text
    return retrieve(
           ^^^^^^^^^
  File "/home/gary.koplik/.conda/envs/py3dep_env/lib/python3.11/site-packages/async_retriever/async_retriever.py", line 394, in retrieve
    inp = BaseRetriever(
          ^^^^^^^^^^^^^^
  File "/home/gary.koplik/.conda/envs/py3dep_env/lib/python3.11/site-packages/async_retriever/_utils.py", line 175, in __init__
    self.url_kwds = self.generate_requests(urls, request_kwds, self.file_paths)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gary.koplik/.conda/envs/py3dep_env/lib/python3.11/site-packages/async_retriever/_utils.py", line 208, in generate_requests
    raise InputValueError(f"request_kwds ({invalids})", list(session_kwds))
async_retriever.exceptions.InputValueError: Given request_kwds (params) is invalid. Valid options are:
self
method
str_or_url
expire_after
kwargs

Anything else we need to know?

For debugging purposes, my weekly pipeline passed on 9/14/23 but failed on 9/21/23, so hopefully that can narrow which dependency issues are to blame.

Environment

PACKAGE VERSION

aiodns 3.0.0
aiohttp 3.8.5
aiohttp-client-cache 0.9.1
aiosqlite 0.19.0
async-retriever 0.15.0
bottleneck N/A
brotli 1.1.0
click 8.1.7
cytoolz 0.12.2
dask 2023.9.2
defusedxml 0.7.1
folium N/A
geopandas 0.14.0
h5netcdf N/A
hydrosignatures N/A
lxml 4.9.3
matplotlib N/A
netCDF4 1.6.4
networkx N/A
numba N/A
numpy 1.26.0
owslib 0.29.2
pandas 2.1.1
py3dep 0.15.0
pyarrow N/A
pydaymet N/A
pygeohydro N/A
pygeoogc 0.15.0
pygeos N/A
pygeoutils 0.15.0
pynhd N/A
pynldas2 N/A
pyproj 3.6.1
pytest N/A
pytest-cov N/A
rasterio 1.3.8
requests 2.31.0
requests-cache 1.1.0
richdem N/A
rioxarray 0.15.0
scipy 1.11.2
shapely 2.0.1
tables N/A
ujson 5.8.0
urllib3 2.0.5
xarray 2023.8.0
xdist N/A
yaml N/A

[cheginit]

Thanks for reporting the issue. I have already addressed it and just released a new version for all HyRiver packages (0.15.2). They are already available on PyPi, so you can install with pip but releasing on conda-forge will probably take a day.

[gjkoplik]

All is working on my end now, thanks!