Imports interfere with AWS `boto3`
Closed this issue · 10 comments
Describe the bug
Does arcgis
monkey patch ssl, or do any non standard import behavior?
Importing acrgis
first creates an error when trying to use boto3
.
One of the two libraries is doing some badly behaved import side effects. I don't know which?
To Reproduce
Simply import FeatureLayer
before trying to import and create a boto3
s3 client.
I realize this is tricky unless you happen to have some AWS profile configured;
import boto3
from arcgis.features import FeatureLayer
session = boto3.Session(
profile_name="..."
)
client = session.client('s3')
# Fails... see error message further below
A workaround is to swap the order of imports:
# Swapping the order of imports; and it works again:
from arcgis.features import FeatureLayer
import boto3
session = boto3.Session(profile_name="...")
client = session.client('s3')
# Succeeds
error:
{
"name": "RecursionError",
"message": "maximum recursion depth exceeded",
"stack": "---------------------------------------------------------------------------
RecursionError Traceback (most recent call last)
c:\\Users\\...\\LOCAL\\GIT\\pta_api\\test_rtop_archive.ipynb Cell 2 line 1
----> <a href='vscode-notebook-cell:/...archive.ipynb#W1sZmlsZQ%3D%3D?line=0'>1</a> pta_api.fetch_rtop_archive(
<a href='vscode-notebook-cell:/...archive.ipynb#W1sZmlsZQ%3D%3D?line=1'>2</a> aws_access_key_id = keyring.get_password(\"...-key\", \"user\"),
<a href='vscode-notebook-cell:/...archive.ipynb#W1sZmlsZQ%3D%3D?line=2'>3</a> aws_secret_access_key = keyring.get_password(\"...-access-key\", \"user\"),
<a href='vscode-notebook-cell:/...archive.ipynb#W1sZmlsZQ%3D%3D?line=3'>4</a> year = 2023,
<a href='vscode-notebook-cell:/...archive.ipynb#W1sZmlsZQ%3D%3D?line=4'>5</a> )
File ~\\LOCAL\\GIT\\pta_api\\src\\pta_api\\_fetch_rtop_archive.py:51, in fetch_rtop_archive(aws_access_key_id, aws_secret_access_key, year, month, day, region_name, parse_dates, parse_geometry)
48 elif day is not None:
49 raise ValueError('day specified without month')
---> 51 client = session.client('s3')
53 # download all files (one by one; not ideal at all, but this is the only way)
54 results = []
File c:\\Users\\...\\AppData\\Local\\miniconda3\\Lib\\site-packages\\boto3\\session.py:299, in Session.client(self, service_name, region_name, api_version, use_ssl, verify, endpoint_url, aws_access_key_id, aws_secret_access_key, aws_session_token, config)
217 def client(
218 self,
219 service_name,
(...)
228 config=None,
229 ):
230 \"\"\"
231 Create a low-level service client by name.
232
(...)
297
298 \"\"\"
--> 299 return self._session.create_client(
300 service_name,
301 region_name=region_name,
302 api_version=api_version,
303 use_ssl=use_ssl,
304 verify=verify,
305 endpoint_url=endpoint_url,
306 aws_access_key_id=aws_access_key_id,
307 aws_secret_access_key=aws_secret_access_key,
308 aws_session_token=aws_session_token,
309 config=config,
310 )
File c:\\Users\\...\\AppData\\Local\\miniconda3\\Lib\\site-packages\\botocore\\session.py:997, in Session.create_client(self, service_name, region_name, api_version, use_ssl, verify, endpoint_url, aws_access_key_id, aws_secret_access_key, aws_session_token, config)
980 self._add_configured_endpoint_provider(
981 client_name=service_name,
982 config_store=config_store,
983 )
985 client_creator = botocore.client.ClientCreator(
986 loader,
987 endpoint_resolver,
(...)
995 user_agent_creator=user_agent_creator,
996 )
--> 997 client = client_creator.create_client(
998 service_name=service_name,
999 region_name=region_name,
1000 is_secure=use_ssl,
1001 endpoint_url=endpoint_url,
1002 verify=verify,
1003 credentials=credentials,
1004 scoped_config=self.get_scoped_config(),
1005 client_config=config,
1006 api_version=api_version,
1007 auth_token=auth_token,
1008 )
1009 monitor = self._get_internal_component('monitor')
1010 if monitor is not None:
File c:\\Users\\...\\AppData\\Local\\miniconda3\\Lib\\site-packages\\botocore\\client.py:159, in ClientCreator.create_client(self, service_name, region_name, is_secure, endpoint_url, verify, credentials, scoped_config, api_version, client_config, auth_token)
146 region_name, client_config = self._normalize_fips_region(
147 region_name, client_config
148 )
149 endpoint_bridge = ClientEndpointBridge(
150 self._endpoint_resolver,
151 scoped_config,
(...)
157 ),
158 )
--> 159 client_args = self._get_client_args(
160 service_model,
161 region_name,
162 is_secure,
163 endpoint_url,
164 verify,
165 credentials,
166 scoped_config,
167 client_config,
168 endpoint_bridge,
169 auth_token,
170 endpoints_ruleset_data,
171 partition_data,
172 )
173 service_client = cls(**client_args)
174 self._register_retries(service_client)
File c:\\Users\\...\\AppData\\Local\\miniconda3\\Lib\\site-packages\\botocore\\client.py:490, in ClientCreator._get_client_args(self, service_model, region_name, is_secure, endpoint_url, verify, credentials, scoped_config, client_config, endpoint_bridge, auth_token, endpoints_ruleset_data, partition_data)
466 def _get_client_args(
467 self,
468 service_model,
(...)
479 partition_data,
480 ):
481 args_creator = ClientArgsCreator(
482 self._event_emitter,
483 self._user_agent,
(...)
488 user_agent_creator=self._user_agent_creator,
489 )
--> 490 return args_creator.get_client_args(
491 service_model,
492 region_name,
493 is_secure,
494 endpoint_url,
495 verify,
496 credentials,
497 scoped_config,
498 client_config,
499 endpoint_bridge,
500 auth_token,
501 endpoints_ruleset_data,
502 partition_data,
503 )
File c:\\Users\\...\\AppData\\Local\\miniconda3\\Lib\\site-packages\\botocore\\args.py:137, in ClientArgsCreator.get_client_args(self, service_model, region_name, is_secure, endpoint_url, verify, credentials, scoped_config, client_config, endpoint_bridge, auth_token, endpoints_ruleset_data, partition_data)
134 new_config = Config(**config_kwargs)
135 endpoint_creator = EndpointCreator(event_emitter)
--> 137 endpoint = endpoint_creator.create_endpoint(
138 service_model,
139 region_name=endpoint_region_name,
140 endpoint_url=endpoint_config['endpoint_url'],
141 verify=verify,
142 response_parser_factory=self._response_parser_factory,
143 max_pool_connections=new_config.max_pool_connections,
144 proxies=new_config.proxies,
145 timeout=(new_config.connect_timeout, new_config.read_timeout),
146 socket_options=socket_options,
147 client_cert=new_config.client_cert,
148 proxies_config=new_config.proxies_config,
149 )
151 serializer = botocore.serialize.create_serializer(
152 protocol, parameter_validation
153 )
154 response_parser = botocore.parsers.create_parser(protocol)
File c:\\Users\\...\\AppData\\Local\\miniconda3\\Lib\\site-packages\\botocore\\endpoint.py:409, in EndpointCreator.create_endpoint(self, service_model, region_name, endpoint_url, verify, response_parser_factory, timeout, max_pool_connections, http_session_cls, proxies, socket_options, client_cert, proxies_config)
406 endpoint_prefix = service_model.endpoint_prefix
408 logger.debug('Setting %s timeout as %s', endpoint_prefix, timeout)
--> 409 http_session = http_session_cls(
410 timeout=timeout,
411 proxies=proxies,
412 verify=self._get_verify_value(verify),
413 max_pool_connections=max_pool_connections,
414 socket_options=socket_options,
415 client_cert=client_cert,
416 proxies_config=proxies_config,
417 )
419 return Endpoint(
420 endpoint_url,
421 endpoint_prefix=endpoint_prefix,
(...)
424 http_session=http_session,
425 )
File c:\\Users\\...\\AppData\\Local\\miniconda3\\Lib\\site-packages\\botocore\\httpsession.py:323, in URLLib3Session.__init__(self, verify, proxies, timeout, max_pool_connections, socket_options, client_cert, proxies_config)
321 self._socket_options = []
322 self._proxy_managers = {}
--> 323 self._manager = PoolManager(**self._get_pool_manager_kwargs())
324 self._manager.pool_classes_by_scheme = self._pool_classes_by_scheme
File c:\\Users\\...\\AppData\\Local\\miniconda3\\Lib\\site-packages\\botocore\\httpsession.py:340, in URLLib3Session._get_pool_manager_kwargs(self, **extra_kwargs)
336 def _get_pool_manager_kwargs(self, **extra_kwargs):
337 pool_manager_kwargs = {
338 'timeout': self._timeout,
339 'maxsize': self._max_pool_connections,
--> 340 'ssl_context': self._get_ssl_context(),
341 'socket_options': self._socket_options,
342 'cert_file': self._cert_file,
343 'key_file': self._key_file,
344 }
345 pool_manager_kwargs.update(**extra_kwargs)
346 return pool_manager_kwargs
File c:\\Users\\...\\AppData\\Local\\miniconda3\\Lib\\site-packages\\botocore\\httpsession.py:349, in URLLib3Session._get_ssl_context(self)
348 def _get_ssl_context(self):
--> 349 return create_urllib3_context()
File c:\\Users\\...\\AppData\\Local\\miniconda3\\Lib\\site-packages\\botocore\\httpsession.py:139, in create_urllib3_context(ssl_version, cert_reqs, options, ciphers)
133 # TLSv1.2 only. Unless set explicitly, do not request tickets.
134 # This may save some bandwidth on wire, and although the ticket is encrypted,
135 # there is a risk associated with it being on wire,
136 # if the server is not rotating its ticketing keys properly.
137 options |= OP_NO_TICKET
--> 139 context.options |= options
141 # Enable post-handshake authentication for TLS 1.3, see GH #1634. PHA is
142 # necessary for conditional client cert authentication with TLS 1.3.
143 # The attribute is None for OpenSSL <= 1.1.0 or does not exist in older
144 # versions of Python. We only enable on Python 3.7.4+ or if certificate
145 # verification is enabled to work around Python issue #37428
146 # See: https://bugs.python.org/issue37428
147 if (
148 cert_reqs == ssl.CERT_REQUIRED or sys.version_info >= (3, 7, 4)
149 ) and getattr(context, \"post_handshake_auth\", None) is not None:
File c:\\Users\\...\\AppData\\Local\\miniconda3\\Lib\\ssl.py:624, in SSLContext.options(self, value)
622 @options.setter
623 def options(self, value):
--> 624 super(SSLContext, SSLContext).options.__set__(self, value)
File c:\\Users\\...\\AppData\\Local\\miniconda3\\Lib\\ssl.py:624, in SSLContext.options(self, value)
622 @options.setter
623 def options(self, value):
--> 624 super(SSLContext, SSLContext).options.__set__(self, value)
[... skipping similar frames: SSLContext.options at line 624 (1478 times)]
File c:\\Users\\...\\AppData\\Local\\miniconda3\\Lib\\ssl.py:624, in SSLContext.options(self, value)
622 @options.setter
623 def options(self, value):
--> 624 super(SSLContext, SSLContext).options.__set__(self, value)
RecursionError: maximum recursion depth exceeded"
}
Expected behavior
It should not matter what order these two libraries are imported.
Platform (please complete the following information):
- OS: Windows
- Python API Version: 2.2.0.1
- Python 3.11.5 (Freshly re-installed to try fix this issue, still doesn't work!)
Additional context
I raised the same issue on the boto3 repo: boto/boto3#3912
@jtroe or @achapkowski Can you advise?
@thehappycheese we do not modify the ssl module at all. It looks like the issue for you is in the botocore library, which powers the boto3 library.
Hi @achapkowski, I added an update to the boto3 ticket. This does seem to be stemming from how arcgis is mutating the ssl module with truststore in Python 3.10+ by calling inject_into_ssl
in your arcgis/auth/api.py
file. I'll start a thread with Seth to see if this can be improved.
Created sethmlarson/truststore#121. After talking with Seth offline, it seems the current usage in arcgis may not be what's intended. We can use that issue for tracking upstream, but the remedy for this will probably require adjustment to the current logic in arcgis.
It may be useful to reopen this to track that portion.
Indeed this will require some action on arcgis, we weren't clear enough in the docs about who should be using inject_into_ssl()
by only hinting at "you must run this asap in your program" and by definition libraries don't control when they run their code since users can rearrange imports arbitrarily and expect things to work.
Arcgis should change its usage from inject_into_ssl
to use truststore.SSLContext
and then the issue will resolve itself.
@sethmlarson we are using truststore for the python 3.10+ version of python, but this issue is for python 3.7-3.9.
I think from the details that were provided above, the original report is for Python 3.11.5.
Python 3.11.5 (Freshly re-installed to try fix this issue, still doesn't work!)
I've been able to reproduce this with just arcgis and urllib3 for all versions of Python 3.10+.
@achapkowski I can also confirm the issue for Python 3.11.7.
Having scratched the surface a bit, the situation does seem much more gnarly, and as I understand it:
- Based on the latest truststore user guide and @sethmlarson's comment above, Truststore advises against using
inject_into_ssl()
- Arcgis's currently uses
requests
as the library to interact with the API (see arcgis/gis/_impl/_con/_connection.py) and Truststore's user guide doesn't provide an example to usetruststore.SSLContext
with requests - This is due to the fact that requests being resistant into allowing users to use their own SSLContext (dates as far back as 2016, see psf/requests#2118, psf/requests#2966)
As an end user of this library and having used arcgis within our Django system, currently the only easy option I have is revert to Python 3.9 for now, though would not be an ideal solution in the long run. Would appreciate some means to have a band-aid hack to maintain at Python 3.11+ if possible.
In terms of how arcgis could improve the codebase, my surface level research would lead to two possible solutions:
- If it were to maintain to use the
requests
library, the recommended path would be to use TransportAdapters (e.g.requests.adapters.HTTPAdapter
- it is to be noted that it may not be fool-proof (see psf/requests#5316) - Possibly need to refactor the code to use another library.
urllib3
perhaps?
Just checking in to see if there's a fix for this. For now downgrading to Python < 3.10 should work? Can this issue be reopened as it hasn't been solved yet?