saleweaver/python-amazon-sp-api

cachetools KeyError from auth/access_token_client.py

chazmcgarvey opened this issue · 6 comments

chazmcgarvey commented

Describe the bug

We occasionally catch KeyError exceptions raised from sp_api/auth/access_token_client.py:50 when calling Reports.create_report.

0: "/venv/lib/python3.12/site-packages/cachetools/__init__.py: 97"
1: "/venv/lib/python3.12/site-packages/cachetools/__init__.py: 70"
2: "/venv/lib/python3.12/site-packages/cachetools/__init__.py: 418"
3: "/venv/lib/python3.12/site-packages/cachetools/__init__.py: 113"
4: "/venv/lib/python3.12/site-packages/cachetools/__init__.py: 366"
5: "/venv/lib/python3.12/site-packages/cachetools/__init__.py: 494"
6: "/venv/lib/python3.12/site-packages/cachetools/__init__.py: 79"
7: "/venv/lib/python3.12/site-packages/cachetools/__init__.py: 423"
8: "/venv/lib/python3.12/site-packages/sp_api/auth/access_token_client.py: 50"
9: "/venv/lib/python3.12/site-packages/sp_api/base/client.py: 70"
10: "/venv/lib/python3.12/site-packages/sp_api/base/client.py: 63"
11: "/venv/lib/python3.12/site-packages/sp_api/base/client.py: 97"
12: "/venv/lib/python3.12/site-packages/sp_api/api/reports/reports.py: 113"
13: "/venv/lib/python3.12/site-packages/sp_api/base/helpers.py: 21"

To Reproduce

I'm not sure, yet, what conditions cause the KeyError.

Expected behavior

Handle the KeyError in the library. If there actually is an exceptional condition, raise a more specific exception.

Desktop (please complete the following information):

  • OS: Linux (cloud)

Additional context

  • Using python-amazon-sp-api 1.4.0.
  • Using cachetools 5.3.2.
  • Using python 3.12.2.
  • We're calling Reports.create_report from another thread (via asyncio.to_thread).
saleweaver commented

Is this a long running environment that's sending the requests to the thread?

chazmcgarvey commented

Is this a long running environment that's sending the requests to the thread?

The process might run for several hours. Never more than a day. The environment itself is ephemeral, lasting as long as the process.

saleweaver commented

That's the problem. The access token is valid for an hour, and if it gets deleted just when you're requesting, the key doesn't exist. I'll have a look if there's an easy fix, I haven't used long running tasks for this in years.

  • 👍1
chazmcgarvey commented

Thanks. If the condition can be handled transparently, great, but even if there was just a more specific exception raised we could catch and know to just retry, that'd work.

saleweaver commented

Yeah, going to identify and clarify it at least 😄

nonfatalexec commented

I think I'm getting the same error. I upgraded my OS which upgraded Python then this started happening. It's 100% reproducible. It was working everyday for the previous year before I upgraded. I tried removing pycache in case that was related, but it made no difference.

$ pip list
Package              Version
-------------------- -----------
boto3                1.34.79
botocore             1.34.79
cachetools           5.3.3
certifi              2024.2.2
charset-normalizer   3.3.2
confuse              2.0.1
idna                 3.6
jmespath             1.0.1
pip                  23.2.1
python-amazon-sp-api 1.4.3
python-dateutil      2.9.0.post0
PyYAML               6.0.1
requests             2.31.0
s3transfer           0.10.1
setuptools           67.7.2
six                  1.16.0
urllib3              1.26.18
wheel                0.40.0

  File "/home/gerald/amazon-sp-api/lib/python3.8/site-packages/cachetools/__init__.py", line 68, in __getitem__
    return self.__data[key]
KeyError: 'access_token_57823aada8f070e62b53ef9039dbabf0'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/gerald/amazon-sp-api/lib/python3.8/site-packages/sp_api/auth/access_token_client.py", line 46, in get_auth
    access_token = cache[cache_key]
  File "/home/gerald/amazon-sp-api/lib/python3.8/site-packages/cachetools/__init__.py", line 418, in __getitem__
    return cache_getitem(self, key)
  File "/home/gerald/amazon-sp-api/lib/python3.8/site-packages/cachetools/__init__.py", line 70, in __getitem__
    return self.__missing__(key)
  File "/home/gerald/amazon-sp-api/lib/python3.8/site-packages/cachetools/__init__.py", line 97, in __missing__
    raise KeyError(key)
KeyError: 'access_token_57823aada8f070e62b53ef9039dbabf0'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./get_inventory.py", line 25, in <module>
    report_request = reports_api.create_report(reportType="GET_FLAT_FILE_OPEN_LISTINGS_DATA", dataStartTime=str(date.today())+"T04:00:00", dataEndTime=str(date.today())+"T04:00:00")
  File "/home/gerald/amazon-sp-api/lib/python3.8/site-packages/sp_api/base/helpers.py", line 21, in wrapper
    return function(*args, **kwargs)
  File "/home/gerald/amazon-sp-api/lib/python3.8/site-packages/sp_api/api/reports/reports.py", line 113, in create_report
    return self._request(kwargs.pop('path'), data=kwargs)
  File "/home/gerald/amazon-sp-api/lib/python3.8/site-packages/sp_api/base/client.py", line 97, in _request
    headers=headers or self.headers,
  File "/home/gerald/amazon-sp-api/lib/python3.8/site-packages/sp_api/base/client.py", line 63, in headers
    'x-amz-access-token': self.restricted_data_token or self.auth.access_token,
  File "/home/gerald/amazon-sp-api/lib/python3.8/site-packages/sp_api/base/client.py", line 70, in auth
    return self._auth.get_auth()
  File "/home/gerald/amazon-sp-api/lib/python3.8/site-packages/sp_api/auth/access_token_client.py", line 49, in get_auth
    access_token = self._request(request_url, self.data, self.headers)
  File "/home/gerald/amazon-sp-api/lib/python3.8/site-packages/sp_api/auth/access_token_client.py", line 35, in _request
    raise AuthorizationError(error_code, error_message, response.status_code)
sp_api.auth.exceptions.AuthorizationError: ('invalid_client', 'Client authentication failed', 401)