api_ca does not default to system cert bundle
nlvw opened this issue · 9 comments
What is the issue ?
When using ARA 1.6.0 with a HTTPS API Server the callback plugin will fail due to a certificate error. It seems like it is not checking the default cert bundle on they system (Fedora 37 in my case) as the following fixes the error
[ara]
api_client = http
api_server = https://ara.example.com
api_ca = /etc/pki/tls/cert.pemWithout api_ca the connection fails with
[WARNING]: Failure using method (v2_playbook_on_start) in callback plugin
(<ansible.plugins.callback.ara_default.CallbackModule object at 0x7f4b47a812d0>):
HTTPSConnectionPool(host='ara.nmsu.edu', port=443): Max retries exceeded with url:
/api/v1/playbooks (Caused by SSLError(SSLCertVerificationError(1, '[SSL:
CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer
certificate (_ssl.c:992)')))
PLAY [Test Ansible Connection] ************************************************************
[WARNING]: Failure using method (v2_playbook_on_play_start) in callback plugin
(<ansible.plugins.callback.ara_default.CallbackModule object at 0x7f4b47a812d0>):
'NoneType' object is not subscriptable
TASK [connection test] ********************************************************************
[WARNING]: Failure using method (v2_playbook_on_task_start) in callback plugin
(<ansible.plugins.callback.ara_default.CallbackModule object at 0x7f4b47a812d0>):
'NoneType' object is not subscriptable
ok: [hpc-wiki-p]
TASK [stop ssh client persistant connection] **********************************************
PLAY RECAP ********************************************************************************
hpc-wiki-p : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
[WARNING]: Failure using method (v2_playbook_on_stats) in callback plugin
(<ansible.plugins.callback.ara_default.CallbackModule object at 0x7f4b47a812d0>):
'NoneType' object is not subscriptable
What should be happening ?
The default ca bundle trusted by the system should be used by default when api_ca is not specified
Software Versions
Ansible = 7.1.0 (Core = 2.14)
OS = Fedora 37
Python = 3.11
ARA = 1.6.0
Install Source = pip
Hi @nlvw and thanks for the issue.
Unless mistaken, the client doesn't specify a path to a CA bundle at all (even a default one) when doing requests unless there is one provided via the configuration so I would expect that to come from python or the requests library.
Out of curiosity, could you use pure python requests to see if it also gets a SSL error?
Something like this (replacing the demo URL with your own) should be sufficient:
> python3 -c 'import requests; print(requests.get("https://demo.recordsansible.org/api/v1/").json())'
{'labels': 'https://demo.recordsansible.org/api/v1/labels', 'playbooks': 'https://demo.recordsansible.org/api/v1/playbooks', 'plays': 'https://demo.recordsansible.org/api/v1/plays', 'tasks': 'https://demo.recordsansible.org/api/v1/tasks', 'hosts': 'https://demo.recordsansible.org/api/v1/hosts', 'latesthosts': 'https://demo.recordsansible.org/api/v1/latesthosts', 'results': 'https://demo.recordsansible.org/api/v1/results', 'files': 'https://demo.recordsansible.org/api/v1/files', 'records': 'https://demo.recordsansible.org/api/v1/records'}
For what it's worth, I am also running on Fedora and do not need to specify ``api_ca` when using the demo and it works with a letsencrypt cert.
I know there's an env variable to set different bundle paths but that shouldn't be necessary.
@dmsimard I guess in your case it's not a self signed certificate. As soon as you are using a self signed certificate you have to give the path to the system ca bundle to the Python requests library, e.g. by simple setting REQUESTS_CA_BUNDLE.
Have to do the same for my Ara installations.
And the path differ between e.g. Ubuntu and Fedora. Thus hard to set a default there.
@hille721 I don't have bandwidth to try and reproduce right now but we have a working integration test to validate that it works for self signed certificates here: https://github.com/ansible-community/ara-collection/pull/57/files and it looks sufficient to specify api_ca.
It hasn't run in a while but I will trigger it to get some fresh results.
I can try and investigate later but in the meantime the job returned successfully: https://ansible.softwarefactory-project.io/zuul/build/50e09cec38da4fafbc5dc57136d8f1fe
Don't get me wrong. Don't see this as a bug or problem. We need to set REQUEST_CA_BUNDLE not only for Ara but for all our Python tools in which the requests library is used and there we have a lot. Thus not a problem for me and that's why I won't spend time for further investigation.
And just to clarify, we don't set any certificate / ca-bundle in the Ara config, but ONLY the REQUEST_CA_BUNDLE environment variable.
So either you have to set api_ca in Ara config which will pass this to requests as verify argument OR you can directly specify the REQUEST_CA_BUNDLE env variable without special ara config. Which makes sense if you are having more tools which are using the requests library
Using requests with not self signed url:
(.venv) $ python3 -c 'import requests; print(requests.get("https://demo.recordsansible.org/api/v1/").json())'
{'labels': 'https://demo.recordsansible.org/api/v1/labels', 'playbooks': 'https://demo.recordsansible.org/api/v1/playbooks', 'plays': 'https://demo.recordsansible.org/api/v1/plays', 'tasks': 'https://demo.recordsansible.org/api/v1/tasks', 'hosts': 'https://demo.recordsansible.org/api/v1/hosts', 'latesthosts': 'https://demo.recordsansible.org/api/v1/latesthosts', 'results': 'https://demo.recordsansible.org/api/v1/results', 'files': 'https://demo.recordsansible.org/api/v1/files', 'records': 'https://demo.recordsansible.org/api/v1/records'}
Using requests with self signed certificate, wont't work
(.venv) $ python3 -c 'import requests; print(requests.get("https://raspi4.fritz.box:9090/"))'
Traceback (most recent call last):
File "/home/hille/.venv/lib64/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/home/hille/.venv/lib64/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/home/hille/.venv/lib64/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
conn.connect()
File "/home/hille/.venv/lib64/python3.10/site-packages/urllib3/connection.py", line 414, in connect
self.sock = ssl_wrap_socket(
File "/home/hille/.venv/lib64/python3.10/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(
File "/home/hille/.venv/lib64/python3.10/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib64/python3.10/ssl.py", line 513, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib64/python3.10/ssl.py", line 1071, in _create
self.do_handshake()
File "/usr/lib64/python3.10/ssl.py", line 1342, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/hille/.venv/lib64/python3.10/site-packages/requests/adapters.py", line 489, in send
resp = conn.urlopen(
File "/home/hille/.venv/lib64/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/home/hille/.venv/lib64/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='raspi4.fritz.box', port=9090): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/hille/.venv/lib64/python3.10/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/home/hille/.venv/lib64/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/home/hille/.venv/lib64/python3.10/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/home/hille/.venv/lib64/python3.10/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/home/hille/.venv/lib64/python3.10/site-packages/requests/adapters.py", line 563, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='raspi4.fritz.box', port=9090): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))
Simple set REQUESTS_CA_BUNDLE and it will work:
(.venv) $ REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt python3 -c 'import requests; print(requests.get("https://raspi4.fritz.box:9090/"))'
<Response [200]>
tested with Fedora 36 and a Virtual Environment.
Fun fact: If I'm using not a virtual environment, but the system Python it will work without setting REQUESTS_CA_BUNDLE.
$ echo $REQUESTS_CA_BUNDLE
$ python3 -c 'import requests; print(requests.get("https://raspi4.fritz.box:9090/"))'
<Response [200]>
$ which python3
/usr/bin/python3
Yes, it's a common enough issue that I have seen with python things before but I had not noticed it with ara (yet?).
I suppose since this is a known issue we should add a little something to the troubleshooting docs: https://github.com/ansible-community/ara/blob/master/doc/source/troubleshooting.rst
I'll leave this issue opened in the meantime.