ESMValTool Tutorial: recipe_python extract_location preprocessor: GeocoderInsufficientPrivileges Error
Closed this issue · 9 comments
Hi All,
I am preparing for an Introduction to ESMValTool workshop and I came across an error while trying to run the recipe_python.
The extract_location preprocessor returns an error and the recipe fails:
"""
Traceback (most recent call last):
File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/geopy/geocoders/base.py", line 368, in _call_geocoder
result = self.adapter.get_json(url, timeout=timeout, headers=req_headers)
File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/geopy/adapters.py", line 447, in get_json
resp = self._request(url, timeout=timeout, headers=headers)
File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/geopy/adapters.py", line 475, in _request
raise AdapterHTTPError(
geopy.adapters.AdapterHTTPError: Non-successful status code 403
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/_task.py", line 816, in _run_task
output_files = task.run()
File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/_task.py", line 264, in run
self.output_files = self._run(input_files)
File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/preprocessor/__init__.py", line 670, in _run
product.apply(step, self.debug)
File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/preprocessor/__init__.py", line 470, in apply
self.cubes = preprocess(self.cubes, step,
File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/preprocessor/__init__.py", line 379, in preprocess
result.append(_run_preproc_function(function, item, settings,
File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/preprocessor/__init__.py", line 328, in _run_preproc_function
return function(items, **kwargs)
File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/preprocessor/_regrid.py", line 377, in extract_location
geolocation = geolocator.geocode(location)
File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/geopy/geocoders/nominatim.py", line 297, in geocode
return self._call_geocoder(url, callback, timeout=timeout)
File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/geopy/geocoders/base.py", line 388, in _call_geocoder
res = self._adapter_error_handler(error)
File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/geopy/geocoders/base.py", line 411, in _adapter_error_handler
raise exc_cls(str(error)) from error
geopy.exc.GeocoderInsufficientPrivileges: Non-successful status code 403
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/_main.py", line 524, in run
fire.Fire(ESMValTool())
File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/_main.py", line 411, in run
self._run(recipe, session)
File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/_main.py", line 453, in _run
process_recipe(recipe_file=recipe, session=session)
File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/_main.py", line 125, in process_recipe
recipe.run()
File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/_recipe/recipe.py", line 1294, in run
self.tasks.run(max_parallel_tasks=self.session['max_parallel_tasks'])
File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/_task.py", line 738, in run
self._run_parallel(address, max_parallel_tasks)
File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/_task.py", line 782, in _run_parallel
_copy_results(task, running[task])
File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/_task.py", line 805, in _copy_results
task.output_files, task.products = future.get()
File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/multiprocessing/pool.py", line 774, in get
raise self._value
geopy.exc.GeocoderInsufficientPrivileges: Non-successful status code 403
I am attaching the main_log_debug.txt
Any help is appreciated as the tutorial day is fast approaching.
I am using geopy v2.3.0 which when running the following code:
from geopy.geocoders import Nominatim
geolocator = Nominatim()returns a ConfigurationError:
ConfigurationError: Using Nominatim with default or sample `user_agent` "geopy/2.3.0" is strongly discouraged, as it violates Nominatim's ToS https://operations.osmfoundation.org/policies/nominatim/ and may possibly cause 403 and 429 HTTP errors. Please specify a custom `user_agent` with `Nominatim(user_agent="my-application")` or by overriding the default `user_agent`: `geopy.geocoders.options.default_user_agent = "my-application"`.
The following works
import geopy
from geopy.geocoders import Nominatim
geopy.geocoders.options.default_user_agent = "esmvaltool"
address = "90, Park Avenue, New York City, New York, 10016"
geolocator = Nominatim()
location = geolocator.geocode(address, timeout=10, exactly_one=False)While the extract_location preprocessor uses something like this:
import geopy
from geopy.geocoders import Nominatim
import ssl
address = "90, Park Avenue, New York City, New York, 10016"
ssl_context = ssl.create_default_context()
geolocator = Nominatim(user_agent="esmvalcore", ssl_context=ssl_context)
location = geolocator.geocode(address, timeout=10, exactly_one=False)And that fails. I tried on a couple of systems and I can reproduce the issue
Actually, the error seems to occur randomly. When it crashes, I can get it to work by changing the user_agent to something else than esmvalcore. Are we hitting some sort of limit on the number of requests?
Anyway, that is annoying.
@rbeucher - I have tried to reproduce the issue you encountered but couldn't get this failure. I have installed the Tool with mamba env create -n esmvaltool esmvaltool python=3.10 to reproduce your environment. Running examples/recipe_python.yml and the last code snippet from this #2171 (comment) work fine for me. I wonder if this issue is similar to the one seen at the MetOffice (see #2012) due to custom installations on clusters with specific restrictions...
As we had issues with this preprocessor in our nightly tests where examples/recipe_python.yml was run frequently, we made a copy of that recipe: testing/recipe_python_for_CI.yml which is identical to the former one but with a hard-coded location (see ESMValGroup/ESMValTool#3157). Perhaps running testing/recipe_python_for_CI.yml could be a temporary workaround for you.
Maybe we should reconsider using the extract_location preprocessor function in the example recipe. It is nice that participants of trainings can change the location, but the service is a bit unreliable and that can be confusing for people new to ESMValTool. @ESMValGroup/userengagementteam Would you have ideas for which preprocessor functions are 'nice' and interesting for the examples/recipe_python.yml recipe?
Maybe we should reconsider using the
extract_locationpreprocessor function in the example recipe. It is nice that participants of trainings can change the location, but the service is a bit unreliable and that can be confusing for people new to ESMValTool. @ESMValGroup/userengagementteam Would you have ideas for which preprocessor functions are 'nice' and interesting for the examples/recipe_python.yml recipe?
@nhsavage thought this might interest you, since you had difficulties with extract_location for CORDEX emample recipe...