SSL3_GET_SERVER_CERTIFICATE: certificate verify failed

[betabank]

Hi, when issuing:
glacier-cmd lsvault
we get:
ssl.SSLError: [Errno 1] _ssl.c:492: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

Open SSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
CentOS 6.8
Python 2.6.6

[kmcg440]

I am getting this same error as well with all glacier-cmd commands. It looks like on June 3rd, 2016 is when this issue started happening for us.

[kmcg440]

I did an upgrade on our server and it fixed the issue. Not sure which package it was that fixed it but here is the list of what was upgraded.

[kitchin]

I still get the error.

# yum update
...
No Packages marked for Update

#  rpm --query centos-release
centos-release-6-8.el6.centos.12.3.x86_64

# openssl version
OpenSSL 1.0.1e-fips 11 Feb 2013

# python --version
Python 2.6.6

# glacier-cmd lsvault
Traceback (most recent call last):
  File "/usr/bin/glacier-cmd", line 9, in <module>
    load_entry_point('glacier==0.2.dev0', 'console_scripts', 'glacier-cmd')()
  File "/usr/lib/python2.6/site-packages/glacier-0.2.dev0-py2.6.egg/glacier/glacier.py", line 929, in main
    args.func(args)
...
  File "/usr/lib/python2.6/site-packages/boto-2.39.0-py2.6.egg/boto/connection.py", line 1030, in _mexe
    raise ex
ssl.SSLError: [Errno 1] _ssl.c:492: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

[kmcg440]

This is an SSL verification issue another workaround that works is to edit /usr/local/lib/python2.7/dist-packages/boto-2.39.0-py2.7.egg/boto/connection.py and set validate_certs=True to False. This will stop the error but is insecure. Its under class AWSAuthConnection(object)

[ebyrne242]

I see this too. CentOS 7.2 with all current updates. I also updated all of the python dependencies installed via pip. Since disabling cert validation in boto works around the problem, I suppose the problem is actually in boto and not glacier-cmd itself. I also tried updating boto's cert list (/usr/lib/python2.7/site-packages/boto/cacerts/cacerts.txt on my system) from the locations listed in the file, with no change.

[Deleted user]

Same, also getting this error.

CentOS Linux release 7.2.1511 (Core) all updates applied.

[coofercat]

I have the same problem with some Centos 7 boxes. The Redhat doco (https://access.redhat.com/articles/2039753) says you can disable cert checking by editing /etc/python/cert-verification.cfg and setting a value to 'false'. On my systemthat was already the case, so doesn't work. The same doco also talks about the ca-certificates RPM package, which in my case is up to date.

I wondered if 'backports.ssl-match-hostname' might be something to do with it. It arrives on the box by RPM, but a pip update of it didn't fix the problem either.

Ultimately, it seems that Python is unable to verify the cert at Amazon (which I think is a 'core' python issue, rather than Boto or glacier-cmd per-se). I'm running out of ideas how to convince Python it's okay, other than to hack in disabling verification, as others have done, though.

[coofercat]

(I also have a Centos 6.8 system affected by this)

Looking in code, in boto/connection.py I can see this:

class AWSAuthConnection(object):
    def __init__(self, host, aws_access_key_id=None,
                 aws_secret_access_key=None,
                 is_secure=True, port=None, proxy=None, proxy_port=None,
                 proxy_user=None, proxy_pass=None, debug=0,
                 https_connection_factory=None, path='/',
                 provider='aws', security_token=None,
                 suppress_consec_slashes=True,
                 validate_certs=True, profile_name=None):

...which tells me that Boto tries to validate certs. Indeed, if I change validate_certs to False, I can communicate with Glacier with no problems. However, I'd rather not hack Boto if I can avoid it.

I've tried to follow the glacier-cmd code through, but honestly need someone with more familiarity to take a look for me. I'd love for this to be a config option, but figuring out how to make the code do that is proving troublesome for me.

[wvmarle]

As I see it, this shouldn't be too hard. This validate_certs parameter has True as default, so all that has to be done is add this as parameter when initialising AWSAuthConnection, something like:

connection = AWSAuthConnection(host, validate_certs=False)

No need to change it in Boto directly, it's done through the call by glacier-cmd, which in turn means it can be added as config option there.
Now it's been years since I've been working on glacier-cmd so completely forgot where and how it's called, would need to search for it.

[fsantiago07044]

i set validate_certs to false to test and mine continues to fail? centos 7.2.x, fully patched. debug log does complain of same sslerror / disconnecting.

[soichih]

Same issue here.

$ aws glacier list-vaults --account-id -
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:765)

I am not sure which hosts it's having a problem with, but I don't see anything obviously wrong from the openssl output when I try to connect to https://glacier.us-east-1.amazonaws.com

$ openssl s_client -connect glacier.us-east-1.amazonaws.com:443 

[ebollens]

To follow up on this:

This is an SSL verification issue another workaround that works is to edit /usr/local/lib/python2.7/dist-packages/boto-2.39.0-py2.7.egg/boto/connection.py and set validate_certs=True to False. This will stop the error but is insecure. Its under class AWSAuthConnection(object)

Instead of modifying Boto, you can add the following:

[Boto]
https_validate_certificates = False

To one of:

• /etc/boto.cfg
• ~/.boto

[coofercat]

I can confirm the config file method works well as an (insecure) workaround on Centos 7 - thanks for the tip!

Now I/we just need to figure out which bit of Python is using whichever out of date certificate store and talk to whichever vendor/provider to get it fixed :-(

[ladamson]

Debian Wheezy: Error did occur, but updating the system removed the error (tested using OpenSSL)
Debian Jessie: Error did occur, but updating the system removed the error

If you experience the error, running

openssl s_client -connect glacier.us-east-1.amazonaws.com:443

will show you which certificate is missing (update the Glacier endpoint if necessary). On Debian Wheezy, before updating:

CONNECTED(00000003)
depth=2 C = US, O = "VeriSign, Inc.", OU = VeriSign Trust Network, OU = "(c) 2006 VeriSign, Inc. - For authorized use only", CN = VeriSign Class 3 Public Primary Certification Authority - G5
verify error:num=20:unable to get local issuer certificate
verify return:0
---

Interestingly, an updated ca-certificates on Debian Wheezy was not necessary to fix the problem. The fix must have been introduced by one of these packages (based on @kmcg440's list above and my updates on Wheezy and Jessie):

apache2-utils
bind9-host
curl
dnsutils
dpkg
dpkg-dev
e2fslibs
gnupg
gpgv
libc6-dev
libc-dev-bin
libpython2.7
libss2
mysql-common
ntpdate
openssh-client
openssh-server
perl
perl-base
perl-modules
php5-cli
php5-common
python2.7
subversion
tcpdump
tzdata
wget

[jamshid]

fwiw validate_certs=False to S3Connection did not work for me on CentOS7.4, neither did the ~/.boto. I had to export PYTHONHTTPSVERIFY=0.