Getting "botocore.exceptions.NoCredentialsError: Unable to locate credentials" on ec2 instance as of botocore-1.13.21
Closed this issue · 3 comments
It seems like as of botocore-1.13.21, using boto3 on my ec2 instances is failing with botocore.exceptions.NoCredentialsError: Unable to locate credentials
where it was working perfectly fine beforehand.
Traceback (most recent call last):
File "<string>", line 16, in <module>
File "/foo/lib64/python3.7/site-packages/botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/foo/lib64/python3.7/site-packages/botocore/client.py", line 648, in _make_api_call
operation_model, request_dict, request_context)
File "/foo/lib64/python3.7/site-packages/botocore/client.py", line 667, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/foo/lib64/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "/foo/lib64/python3.7/site-packages/botocore/endpoint.py", line 132, in _send_request
request = self.create_request(request_dict, operation_model)
File "/foo/lib64/python3.7/site-packages/botocore/endpoint.py", line 116, in create_request
operation_name=operation_model.name)
File "/foo/lib64/python3.7/site-packages/botocore/hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/foo/lib64/python3.7/site-packages/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/foo/lib64/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "/foo/lib64/python3.7/site-packages/botocore/signers.py", line 90, in handler
return self.sign(operation_name, request)
File "/foo/lib64/python3.7/site-packages/botocore/signers.py", line 157, in sign
auth.add_auth(request)
File "/foo/lib64/python3.7/site-packages/botocore/auth.py", line 357, in add_auth
raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials
The code itself is pretty simple:
import boto3
region = "foo"
id = 123456789
secret_name = "bar"
asm = boto3.client(
service_name='secretsmanager',
region_name=region
)
secret = asm.get_secret_value(
SecretId=f'arn:aws:secretsmanager:{region}:{id}:secret:{secret_name}'
)
I'm experiencing similar. When creating a boto3 client on an EKS node I get the following boto3 logging output:
`
2019-11-20 14:58:56,328 botocore.credentials [DEBUG] Looking for credentials via: iam-role
2019-11-20 14:58:57,334 botocore.utils [DEBUG] Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/api/token: Read timeout on endpoint URL: "http://169.254.169.254/latest/api/token"
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 416, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib64/python3.7/http/client.py", line 1336, in getresponse
response.begin()
File "/usr/lib64/python3.7/http/client.py", line 306, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.7/http/client.py", line 267, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib64/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/botocore/httpsession.py", line 263, in send
chunked=self._chunked(request.headers),
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 376, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
raise value
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
chunked=chunked,
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 423, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 331, in _raise_timeout
self, url, "Read timed out. (read timeout=%s)" % timeout_value
urllib3.exceptions.ReadTimeoutError: AWSHTTPConnectionPool(host='169.254.169.254', port=80): Read timed out. (read timeout=1)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/botocore/utils.py", line 295, in _fetch_metadata_token
response = self._session.send(request.prepare())
File "/usr/local/lib/python3.7/site-packages/botocore/httpsession.py", line 289, in send
raise ReadTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL: "http://169.254.169.254/latest/api/token"
`
The IMDSv2 metadata token service on this node does not appear functional. The following request just hangs:
curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"
botocore==1.13.19 however does work in this environment, presumably using IMDSv1 (which is functional in this env.
This has been confirmed and has been fixed as of 1.13.22.
See the below issue.
Duplicate of #1892
Ah yes my mistake, I didn't think to look in the closed issues.