Getting "botocore.exceptions.NoCredentialsError: Unable to locate credentials" on ec2 instance as of botocore-1.13.21

[fchorney]

It seems like as of botocore-1.13.21, using boto3 on my ec2 instances is failing with botocore.exceptions.NoCredentialsError: Unable to locate credentials where it was working perfectly fine beforehand.

Traceback (most recent call last):
  File "<string>", line 16, in <module>
  File "/foo/lib64/python3.7/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/foo/lib64/python3.7/site-packages/botocore/client.py", line 648, in _make_api_call
    operation_model, request_dict, request_context)
  File "/foo/lib64/python3.7/site-packages/botocore/client.py", line 667, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/foo/lib64/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/foo/lib64/python3.7/site-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/foo/lib64/python3.7/site-packages/botocore/endpoint.py", line 116, in create_request
    operation_name=operation_model.name)
  File "/foo/lib64/python3.7/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/foo/lib64/python3.7/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/foo/lib64/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/foo/lib64/python3.7/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/foo/lib64/python3.7/site-packages/botocore/signers.py", line 157, in sign
    auth.add_auth(request)
  File "/foo/lib64/python3.7/site-packages/botocore/auth.py", line 357, in add_auth
    raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials

The code itself is pretty simple:

import boto3
region = "foo"
id = 123456789
secret_name = "bar"
asm = boto3.client(
    service_name='secretsmanager',
    region_name=region
)
secret = asm.get_secret_value(
    SecretId=f'arn:aws:secretsmanager:{region}:{id}:secret:{secret_name}'
)

[jweite]

I'm experiencing similar. When creating a boto3 client on an EKS node I get the following boto3 logging output:

`
2019-11-20 14:58:56,328 botocore.credentials [DEBUG] Looking for credentials via: iam-role
2019-11-20 14:58:57,334 botocore.utils [DEBUG] Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/api/token: Read timeout on endpoint URL: "http://169.254.169.254/latest/api/token"
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 416, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib64/python3.7/http/client.py", line 1336, in getresponse
response.begin()
File "/usr/lib64/python3.7/http/client.py", line 306, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.7/http/client.py", line 267, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib64/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/botocore/httpsession.py", line 263, in send
chunked=self._chunked(request.headers),
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 376, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
raise value
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
chunked=chunked,
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 423, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 331, in _raise_timeout
self, url, "Read timed out. (read timeout=%s)" % timeout_value
urllib3.exceptions.ReadTimeoutError: AWSHTTPConnectionPool(host='169.254.169.254', port=80): Read timed out. (read timeout=1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/botocore/utils.py", line 295, in _fetch_metadata_token
response = self._session.send(request.prepare())
File "/usr/local/lib/python3.7/site-packages/botocore/httpsession.py", line 289, in send
raise ReadTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL: "http://169.254.169.254/latest/api/token"
`

The IMDSv2 metadata token service on this node does not appear functional. The following request just hangs:
curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"

botocore==1.13.19 however does work in this environment, presumably using IMDSv1 (which is functional in this env.

[joguSD]

This has been confirmed and has been fixed as of 1.13.22.
See the below issue.
Duplicate of #1892

[fchorney]

Ah yes my mistake, I didn't think to look in the closed issues.