pyca/pyopenssl

invalid pointer with cryptography 37.0.3

Closed this issue · 5 comments

alexrashed commented

We are currently facing issues with the latest release of cryptography (37.0.3, released yesterday - 2022-06-21) in combination with the latest version of pyOpenSSL (22.0.0).
When executing our unit tests on CircleCI, there is an invalid pointer access when signing a cert using pyOpenSSL:

tests/unit/utils/test_common.py::test_generate_ssl_cert munmap_chunk(): invalid pointer
Fatal Python error: Aborted

Current thread 0x00007f549027b740 (most recent call first):
  File "/tmp/workspace/repo/.venv/lib/python3.10/site-packages/OpenSSL/crypto.py", line 1229 in sign
  File "/tmp/workspace/repo/localstack/utils/crypto.py", line 114 in generate_ssl_cert
  File "/tmp/workspace/repo/localstack/utils/sync.py", line 99 in _wrapper
  File "/tmp/workspace/repo/tests/unit/utils/test_common.py", line 251 in test_generate_ssl_cert
  ...

Extension modules: _cffi_backend, markupsafe._speedups, yaml._yaml, psutil._psutil_linux, psutil._psutil_posix (total: 5)
Aborted (core dumped)

Our CircleCI runs are public, an affected run can be seen here.

I created a minimum reproducible example:

from OpenSSL import crypto
k = crypto.PKey()
k.generate_key(crypto.TYPE_RSA, 2048)
cert = crypto.X509()
cert.set_pubkey(k)
cert.sign(k, "SHA256")

Unfortunately, I cannot reproduce this issue locally, but it is reproducible in CircleCI's Ubuntu images (both the latest 20.04 and 22.04):

(.venv) circleci@xxx:/tmp/workspace/repo$ cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
(.venv) circleci@xxx:/tmp/workspace/repo$ uname -a
Linux ip-172-28-92-14 5.15.0-1009-aws #11-Ubuntu SMP Thu May 26 19:34:47 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
(.venv) circleci@xxx/tmp/workspace/repo$ openssl version
OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)
(.venv) circleci@xxx:/tmp/workspace/repo$ python -VV
Python 3.10.3 (main, Jun  6 2022, 19:11:08) [GCC 11.2.0]
(.venv) circleci@xxx:/tmp/workspace/repo$ pip freeze | grep -E 'cryptography|pyOpenSSL'
cryptography==37.0.2
pyOpenSSL==22.0.0
(.venv) circleci@xxx:/tmp/workspace/repo$ python
Python 3.10.3 (main, Jun  6 2022, 19:11:08) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from OpenSSL import crypto
>>> k = crypto.PKey()
>>> k.generate_key(crypto.TYPE_RSA, 2048)
>>> cert = crypto.X509()
>>> cert.set_pubkey(k)
>>> cert.sign(k, "SHA256")
>>> 
(.venv) circleci@xxx:/tmp/workspace/repo$ pip install --upgrade cryptography==37.0.3
Collecting cryptography==37.0.3
  Using cached cryptography-37.0.3-cp36-abi3-manylinux_2_24_x86_64.whl (4.0 MB)
Requirement already satisfied: cffi>=1.12 in ./.venv/lib/python3.10/site-packages (from cryptography==37.0.3) (1.15.0)
Requirement already satisfied: pycparser in ./.venv/lib/python3.10/site-packages (from cffi>=1.12->cryptography==37.0.3) (2.21)
Installing collected packages: cryptography
  Attempting uninstall: cryptography
    Found existing installation: cryptography 37.0.2
    Uninstalling cryptography-37.0.2:
      Successfully uninstalled cryptography-37.0.2
Successfully installed cryptography-37.0.3
(.venv) circleci@xxx:/tmp/workspace/repo$ pip freeze | grep -E 'cryptography|pyOpenSSL'
cryptography==37.0.3
pyOpenSSL==22.0.0
(.venv) circleci@xxx:/tmp/workspace/repo$ python
Python 3.10.3 (main, Jun  6 2022, 19:11:08) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from OpenSSL import crypto
>>> k = crypto.PKey()
>>> k.generate_key(crypto.TYPE_RSA, 2048)
>>> cert = crypto.X509()
>>> cert.set_pubkey(k)
>>> cert.sign(k, "SHA256")
munmap_chunk(): invalid pointer
Aborted (core dumped)
(.venv) circleci@xxx:/tmp/workspace/repo$

With cryptography==37.0.3 the test constantly fails, with cryptography==37.0.2 the test (and the snippet above) works correctly.
To be honest, I'm not sure if it's caused by an incompatibility of pyopenssl with the latest release of cryptography, or if it's an issue with cryptography itself.
Let me know if I should file an issue in the cryptography repo insead, or if I can help with additional info or tests.

alex commented

Thanks for reporting this. 37.0.3 should be nothing but a new version of OpenSSL in our wheels, so either we screwed up our release procedure, or this is an OpenSSL bug.

alex commented

I don't suppose you know if there's a docker image or some other way to reproduce the CircleCI env easily?

alex commented

Seems very likely that this is: openssl/openssl#18625

I assume OpenSSL will issue a new security release very shortly fixing this. I hope.

alexrashed commented

I don't suppose you know if there's a docker image or some other way to reproduce the CircleCI env easily?

Unfortunately, I don't think so. But CircleCI provides the possibility to SSH into the machine.
I could prepare a simple CircleCI pipeline with the example code if that would be helpful. You could then fork it, connect it to circleci, trigger a build, enable the SSH, and then connect to it...

Seems very likely that this is: openssl/openssl#18625

Thanks for the info!

alex commented

Ok, we have yanked the 37.0.3 due to this regression. Hopefully in a day or two we'll issue a 37.0.4. Going to close this since it should be resolved for now.

  • 👍1