Harvest SSL problem

[davidread]

Harvest of: https://contractsfinder.service.xgov.uk

returns error:

Could not get content because a connection error occurred. [Errno bad handshake] [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')]

[davidread]

Failing test:

$ python -c 'import requests; requests.get("https://contractsfinder.service.xgov.uk/")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/co/ckan/local/lib/python2.7/site-packages/requests/api.py", line 69, in get
    return request('get', url, params=params, **kwargs)
  File "/home/co/ckan/local/lib/python2.7/site-packages/requests/api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/home/co/ckan/local/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/co/ckan/local/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/home/co/ckan/local/lib/python2.7/site-packages/requests/adapters.py", line 431, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [Errno bad handshake] [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')]

It works if we do verify=False

It still fails if I upgrade python modules: certifi pyOpenSSL ndg-httpsclient pyasn1

It still fails if I download and specify latest certs extracted from Mozilla: http://stackoverflow.com/a/12865159

It still fails if we try python 2.7.11 with latest requests version (thanks Ross).

It fails with the same error with curl and wget.

But according to this: https://cryptoreport.websecurity.symantec.com/checker/#certChecker
the site is missing an intermediate certificate. Apparently browsers get around this by shipping with lots of intermediate certificates or downloading it somehow