botocore.exceptions.SSLError: SSL validation failed for <s3 file> [Errno 2] No such file or directory

Question

botocore.exceptions.SSLError: SSL validation failed for <s3 file> [Errno 2] No such file or directory

Opened this issue 4 years ago · 12 comments

maheshmadhusudanan commented 4 years ago

Getting the below error while trying to access remote_env from an s3 bucket

[1592935276008] [DEBUG] 2020-06-23T18:01:16.8Z b8374974-f820-484a-bcc3-64a530712769 Exception received when sending HTTP request.
Traceback (most recent call last):
  File "/var/task/urllib3/util/ssl_.py", line 336, in ssl_wrap_socket
  context.load_verify_locations(ca_certs, ca_cert_dir)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/runtime/botocore/httpsession.py", line 254, in send
  urllib_response = conn.urlopen(
  File "/var/task/urllib3/connectionpool.py", line 719, in urlopen
  retries = retries.increment(
  File "/var/task/urllib3/util/retry.py", line 376, in increment
  raise six.reraise(type(error), error, _stacktrace)
  File "/var/task/six.py", line 703, in reraise
  raise value
  File "/var/task/urllib3/connectionpool.py", line 665, in urlopen
  httplib_response = self._make_request(
  File "/var/task/urllib3/connectionpool.py", line 376, in _make_request
  self._validate_conn(conn)
  File "/var/task/urllib3/connectionpool.py", line 996, in _validate_conn
  conn.connect()
  File "/var/task/urllib3/connection.py", line 352, in connect
  self.sock = ssl_wrap_socket(
  File "/var/task/urllib3/util/ssl_.py", line 338, in ssl_wrap_socket
  raise SSLError(e)
urllib3.exceptions.SSLError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/runtime/botocore/endpoint.py", line 200, in _do_get_response
  http_response = self._send(request)
  File "/var/runtime/botocore/endpoint.py", line 244, in _send
  return self.http_session.send(request)
  File "/var/runtime/botocore/httpsession.py", line 281, in send
  raise SSLError(endpoint_url=request.url, error=e)
botocore.exceptions.SSLError: SSL validation failed for .......  [Errno 2] No such file or directory

My Environment

Zappa version used: 0.51.0
Operating System and Python version: Ubuntu , Python 3.8
Output of pip freeze

appdirs==1.4.3
argcomplete==1.11.1
boto3==1.14.8
botocore==1.17.8
CacheControl==0.12.6
certifi==2019.11.28
cffi==1.14.0
cfn-flip==1.2.3
chardet==3.0.4
click==7.1.2
colorama==0.4.3
contextlib2==0.6.0
cryptography==2.9.2
distlib==0.3.0
distro==1.4.0
docutils==0.15.2
durationpy==0.5
Flask==1.1.2
Flask-Cors==3.0.8
future==0.18.2
h11==0.9.0
hjson==3.0.1
html5lib==1.0.1
httptools==0.1.1
idna==2.8
ipaddr==2.2.0
itsdangerous==1.1.0
Jinja2==2.11.2
jmespath==0.10.0
kappa==0.6.0
lockfile==0.12.2
mangum==0.9.2
MarkupSafe==1.1.1
msgpack==0.6.2
packaging==20.3
pep517==0.8.2
pip-tools==5.2.1
placebo==0.9.0
progress==1.5
pycparser==2.20
pydantic==1.5.1
PyMySQL==0.9.3
pyOpenSSL==19.1.0
pyparsing==2.4.6
python-dateutil==2.6.1
python-slugify==4.0.0
pytoml==0.1.21
PyYAML==5.3.1
requests==2.22.0
retrying==1.3.3
s3transfer==0.3.3
six==1.14.0
starlette==0.13.4
text-unidecode==1.3
toml==0.10.1
tqdm==4.46.1
troposphere==2.6.1
typing-extensions==3.7.4.2
urllib3==1.25.8
uvloop==0.14.0
webencodings==0.5.1
websockets==8.1
Werkzeug==0.16.1
wsgi-request-logger==0.4.6
zappa==0.51.0

Your `zappa_settings.json`:

{
    "dev": {
        "app_function": "main.app",
        "aws_region": "us-west-2",
        "profile_name": "default",
        "project_name": "d3c",
        "runtime": "python3.8",
        "keep_warm":false,
        "cors": true,
        "s3_bucket": "rnd-lambda-deployables",
        "remote_env":"<my remote s3 file>"
    }
}

I have confirmed that my S3 file is accessible from my local ubuntu machine however does not work on aws

Answer 1 · 2020-07-01T16:09:50.000Z

Hey I am getting a similar error. I've setup a config and credentials file in the default path ~/.aws/files, but still didn't work.

Answer 2 · 2020-07-16T13:43:39.000Z

Possibly, packaged botocore version 1.14.8 has got too old?

After a Zappa redeployment in another region, I am getting now this message in the zappa tail:

botocore.exceptions.SSLError: SSL validation failed for https://sts.amazonaws.com/ [Errno 2] No such file or directory

Further observations so far:

This has been observed in the botocore project: aws/aws-cli#2690; interestingly, people talk their about local system fixes. So maybe it's about a path problem in the container environment in AWS? Because locally this problem does not appear, my code gets further until actual STS permissions checking.
Current latest botocore release is 1.17.21

I will try to deploy back to previous region, theoretically environments could differ. UPD: same issue on previous region.

Assumption: AWS has updated Lambda execution environment which triggers that botocore bug.

Answer 3 · 2020-08-29T01:53:09.000Z

This seems to provide a temporary Fix

https://stackoverflow.com/a/63642707/10403203

Answer 4 · 2020-10-22T16:52:03.000Z

I have the same error with command: zappa manage development "collectstatic --noinput" using django-storages.... very weird stuff

Answer 5 · 2020-10-22T16:56:54.000Z

I have the same error with command: zappa manage development "collectstatic --noinput" using django-storages.... very weird stuff

Check out this StackOverflow answer
#2127 (comment)

Answer 6 · 2020-10-22T17:00:04.000Z

@jaywonder20 I saw that... In fact, I changed that part in my code with no solution yet

boto3.client('s3', verify=False, config=Config(signature_version='s3v4'))

Answer 7 · 2020-10-22T17:05:24.000Z

@adosaa did you copy the cacert.pem file and set the REQUESTS_CA_BUNDLE environment variable

Answer 8 · 2020-10-23T19:22:39.000Z

Yep it's works @jaywonder20 ! putting cacert.pem in the root of the project and pointing it with the env. variable, a bit ugly but functional.

Answer 9 · 2020-10-24T00:18:39.000Z

Great @adosaa it a bit ugly but that's a workaround until a better option is found

Answer 10 · 2020-10-31T04:56:18.000Z

I tried the solution they propose but it gives me the following error:

[ERROR] ClientError: An error occurred (InvalidToken) when calling the PutObject operation: The provided token is malformed or otherwise invalid.

This is my condig:

{
    "dev": {
        "aws_region": "us-east-1",
        "django_settings": "my_app.settings",
        "profile_name": "default",
        "project_name": "my_app",
        "runtime": "python3.8",
        "s3_bucket": "zappa-my_app",
        "environment_variables": {
            "USE_S3": "TRUE",
            "AWS_ACCESS_KEY_ID": "***",
            "AWS_SECRET_ACCESS_KEY": "***",
            "AWS_STORAGE_BUCKET_NAME": "my_app",
            "REQUESTS_CA_BUNDLE": "/var/task/cacert.pem"
        }
    }
}

Answer 11 · 2020-11-16T20:19:43.000Z

Also experiencing a similar issue with Zappa 0.51.0 and python 3.6:

botocore.exceptions.SSLError: SSL validation failed for https://secretsmanager.us-east-1.amazonaws.com/ [Errno 2] No such file or directory

[packages] boto3 = "==1.16.7" botocore = "==1.19.17" psycopg2-binary = "==2.8.5" redis = "==3.5.2" zappa = "==0.51.0"

My temporary workaround was to disable SSL verification on the boto call.

verify=false

Answer 12 · 2021-02-05T14:35:48.000Z

I had the same problem both with django-storages and django-s3-storage. @jaywonder20 's solution with the cacert.pem file (see stack overflow) solved the issue for me for now.

My Environment

Your zappa_settings.json:

Your `zappa_settings.json`: