Python 3.12 support
upekkha opened this issue · 25 comments
Python 3.12 has removed the imp module. It was deprecated since Python 3.7, but the deprecation warning ignored by mitogen since 2018 (c2c7caa).
The suggested path forward would be to switch to using importlib instead. Some correspondences are easy (eg imp.get_magic() -> importlib.util.MAGIC_NUMBER), but others look non-trivial and require extra care to keep supporting old Python versions.
I also quickly tried the hack of copying the old imp.py from Python 3.11 into mitogen/compat and basically replacing import imp with import mitogen.compat.imp as imp. This actually allowed me to import mitogen locally without any error messages, but running an ansible playbook fails with
"msg": "EOF on stream; last 100 lines received:\nMITO000\nMITO001\nTraceback (most recent call last):\n File \"<stdin>\", line 61, in <module>\nModuleNotFoundError: No module named 'mitogen'",
Let's hope the mitogen devs find some time looking into this to keep the project alive with Python 3.12+.
At the moment, python 3.12 is only supported on ansible-core 2.16.
https://docs.ansible.com/ansible/latest/reference_appendices/release_and_maintenance.html#ansible-core-release-cycle
Agreed, but despite the lack of official support of Python 3.12, I had no problems running various playbooks with Python3.12 and ansible-core 2.14, 2.15 or 2.16 - but unfortunately only without mitogen.
but others look non-trivial and require extra care to keep supporting old Python versions.
Yeah, this won't be trivial.
Good news.
I had a bit of time today to spend on this, and after 8 hours, got it working.
I managed to make mitogen work with Python 3.12. And with Ansible 2.16.2 as a bonus.
Ansible 2.16.x requires Python 3.8 (due to use of / in argument lists, aka "Positional-only arguments", in ansible.utils.unsafe_proxy module). Not sure if this is documented by ansible. But it is the fact. And some of the hosts I manage do have Python 3.6.x, so that was not nice. But it was very trivial to fix, and it now work on target hosts with Python 3.6. Documentation says 3.6 is supported on a target, but that is not true. 3.8 is required on a target.
Verified configurations:
Host:
- ansible-core 2.16.2, Python 3.12.1, Fedora 39
Targets:
- Python 3.6.8, Centos 8
- Python 3.9.16, Centos 9
- Python 3.9.17, Centos 9
- Python 3.9.18, Centos 9
- Python 3.11.6, Fedora 38
- Python 3.12.0, Fedora 39
- Python 3.12.1, Fedora 39
And as expected, it works smoothly. On a high latency link (225ms), with medium size playbooks (55 tasks), it finishes in 84 seconds vs 169 seconds without mitogen.
I will do some more testing with Host being ansible 2.14.x, 2.15.x, and host using Python 3.6, 3.9, 3.11.
I do not have any older machines anymore, but I can probably spin-up some VM for a test.
The required modifications are a bit of ad-hoc, but are not too big but I should clean it up, and wrap in various conditions and try blocks, so it also works with Python 3.3 and older.
$ git diff --stat
ansible_mitogen/loaders.py | 2 +-
ansible_mitogen/plugins/connection/mitogen_ssh.py | 2 ++
ansible_mitogen/process.py | 12 +++++++++---
ansible_mitogen/strategy.py | 4 +++-
ansible_mitogen/transport_config.py | 14 ++++++++------
mitogen/core.py | 144 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-------
mitogen/master.py | 27 +++++++++++++++++++++------
7 files changed, 181 insertions(+), 24 deletions(-)
$
(30% of this is extra debugging, that was added to help with fixing the issue; and another 10% are some patches from unmerged MRs, that I incorporated just in case)
Attaching a draft patch version for now, in case, I forgot about this.
mitogen-python312.patch.txt
Ok, I found few failing roles with my patch, so still requires some work.
docker_container role failing:
ERROR! [mux 3104762] 14:09:16.442316 E mitogen.[ssh.foobar.example.com.io.sudo.root]: while importing 'numpy.version'
Traceback (most recent call last):
File "<stdin>", line 1671, in exec_module
File "master:/usr/lib64/python3.12/site-packages/numpy/version.py", line 1
SyntaxError: future feature annotations is not defined
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: SyntaxError: future feature annotations is not defined
This is with a target being Python 3.6.8, Centos 8.
It is a bit strange, because I have docker python packages for docker on the target host, and it works (tested in the python interpreter there). There is no numpy, but it does not look like it needs numpy either.
Tracing import docker on Fedora 39, Python 3.11.2, it loads about 466 extra modules, but there is no numpy in the trace (despite this host machine having numpy available), so I am not sure why something is trying to access numpy.
More details about the docker_container role failing.
It looks like docker package pulls websocket package, which optionally depends on numpy. numpy package is not on a target machine, but mitogen module sender sends it anyway (mitogen proactively tries to send all modules that a given module references, even if that reference is under a conditional, try-except block, function or a class - i.e. even if not required, or intended to be loaded lazily).
This causes a minor problem here.
The full traceback is:
Traceback (most recent call last):
File "master:/home/xxx/.local/lib/python3.12/site-packages/mitogen-0.3.5.dev0-py3.12.egg/ansible_mitogen/runner.py", line 973, in _run
self._run_code(code, mod)
File "master:/home/xxx/.local/lib/python3.12/site-packages/mitogen-0.3.5.dev0-py3.12.egg/ansible_mitogen/runner.py", line 937, in _run_code
exec(code, vars(mod))
File "master:/home/witek/.ansible/collections/ansible_collections/community/docker/plugins/modules/docker_container.py", line 1202, in <module>
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<stdin>", line 1678, in exec_module
File "master:/home/witek/.ansible/collections/ansible_collections/community/docker/plugins/module_utils/common.py", line 33, in <module>
from docker import __version__ as docker_version
File "/usr/local/lib/python3.6/site-packages/docker/__init__.py", line 2, in <module>
from .api import APIClient
File "/usr/local/lib/python3.6/site-packages/docker/api/__init__.py", line 2, in <module>
from .client import APIClient
File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 8, in <module>
import websocket
File "/usr/local/lib/python3.6/site-packages/websocket/__init__.py", line 21, in <module>
from ._abnf import *
File "/usr/local/lib/python3.6/site-packages/websocket/_abnf.py", line 37, in <module>
import numpy
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<stdin>", line 1678, in exec_module
File "master:/usr/lib64/python3.12/site-packages/numpy/__init__.py", line 141, in <module>
from . import core
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<stdin>", line 1678, in exec_module
File "master:/usr/lib64/python3.12/site-packages/numpy/core/__init__.py", line 9, in <module>
from numpy.version import version as __version__
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<stdin>", line 1671, in exec_module
File "master:/usr/lib64/python3.12/site-packages/numpy/version.py", line 1
SyntaxError: future feature annotations is not defined
Indeed, that does not work on Python 3.6:
$ python3
Python 3.6.8 (default, Nov 30 2023, 08:04:29)
[GCC 8.5.0 20210514 (Red Hat 8.5.0-21)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from __future__ import annotations
File "<stdin>", line 1
SyntaxError: future feature annotations is not defined
>>>
This is numpy 1.24.4 (from fedora repos: python3-numpy-1.24.4-2.fc39.x86_64, not from pypi)
I will contact numpy devs, to fix this maybe. (This source files, numpy/version.py, does use one annotation, but it is not super critical - the dict[...], can easily be replaced with typing.Dict[...] instead, which should work with a bit older Python, or removed, to allow even older Python). websocket package (aka websocket-client on Pypi) uses numpy.xor... to accelerate websocket mask key xor operation to mask and unmask data (it is a silly feature of websocket unfortunately).
For now workaround is to switch to typing.Dict in the numpy.version.
All that said, websocket-client, actually remove that usage of numpy for mask key xor acceleration about 2 years ago (probably because it is heavy dependency, and probably because the speed improvement is not significant outside of microbenchmarks) - in websocket-client/websocket-client@a462d45 , so that is nice. But, this year, this package started adding typing annotations, starting in websocket-client/websocket-client@86ad0c4 , (in Python 3.9+ format using native type generics), so that is not going to help a lot in the long run to support for example Python 3.6.
Another option is to change this (/usr/local/lib/python3.6/site-packages/websocket/_abnf.py):
try:
if six.PY3:
import numpy
else:
numpy = None
except ImportError:
numpy = None
to also catch SyntaxError exception, not just ImportError, and additionally use numpy = __import__("numpy") to make mitogen fetch it leazily. This is easy, and can be accepted, but considering websocket-client already removed support for numpy, and other projects are unlikely to cherry pick some specific version, this is not practical for the maintainers and will not really achieve anything.
But I will also check my patches, why we send our version of docker / websocket packages via mitogen, if the docker package (possibly at different version tho), exist natively on a target already.
Fortunately for now, there is an easy workaround, and it should affect very few roles / packages.
Updated and slightly improved patch (to not use zombie-imp package even if present, and instead only use importlib if present).
@stefanor Yes, absolutely will make a PR. Need to test it a bit more. (I do not run ansible every day, but I do run few times a week). Tested on two more plays today, all working perfectly. I will try to send PR this week.
Found issue with this patch for mitogen, tested with ansible 2.15.8 (python 3.9.3) and ansible 2.16.2 (python 2.11.6) and destination host Python 3.9.18 CentOS Stream release 9, aws collections:
amazon.aws 6.3.0
community.aws 6.3.0
and
amazon.aws 7.0.0
community.aws 7.0.0
Task
- name: Copy TLS certificates from s3
aws_s3:
aws_access_key: "{{ s3_backup_access_key }}"
aws_secret_key: "{{ s3_backup_secret_key }}"
s3_url: https://ams3.digitaloceanspaces.com
Error:
ansible_2.16.2_aws_error.txt
If to install on target dnf install python-certifi or pip install certifi, that solves issue, but without mitogen it works without any install on target.
@kzinas-adv I see. Thanks for the info. It looks like it tries to load cacert.pem (to validate s3 SSL certificates) as a package resource. It is not very common usage of imports in Python, but definitively something worth adding support for. I will see if I can add a support for this.
Tried with python3.12 (Fedora 38 made with virtual env)
ERROR! Unexpected Exception, this is probably a bug: No module named 'imp'
the full traceback was:
Traceback (most recent call last):
File "/home/domas/.local/lib64/python3.12/site-packages/ansible/cli/__init__.py", line 659, in cli_executor
exit_code = cli.run()
^^^^^^^^^
File "/home/domas/.local/lib64/python3.12/site-packages/ansible/cli/playbook.py", line 156, in run
results = pbex.run()
^^^^^^^^^^
File "/home/domas/.local/lib64/python3.12/site-packages/ansible/executor/playbook_executor.py", line 190, in run
result = self._tqm.run(play=play)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/domas/.local/lib64/python3.12/site-packages/ansible/executor/task_queue_manager.py", line 324, in run
strategy = strategy_loader.get(new_play.strategy, self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/domas/.local/lib64/python3.12/site-packages/ansible/plugins/loader.py", line 864, in get
return self.get_with_context(name, *args, **kwargs).object
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/domas/.local/lib64/python3.12/site-packages/ansible/plugins/loader.py", line 899, in get_with_context
self._module_cache[path] = self._load_module_source(resolved_type_name, path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/domas/.local/lib64/python3.12/site-packages/ansible/plugins/loader.py", line 837, in _load_module_source
spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 994, in exec_module
File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
File "/home/domas/.local/lib/python3.12/site-packages/ansible_mitogen/plugins/strategy/mitogen_linear.py", line 58, in <module>
import ansible_mitogen.strategy
File "/home/domas/.local/lib64/python3.12/site-packages/ansible_mitogen/strategy.py", line 44, in <module>
import ansible_mitogen.mixins
File "/home/domas/.local/lib64/python3.12/site-packages/ansible_mitogen/mixins.py", line 53, in <module>
import mitogen.utils
File "/home/domas/.local/lib64/python3.12/site-packages/mitogen/utils.py", line 38, in <module>
import mitogen.master
File "/home/domas/.local/lib64/python3.12/site-packages/mitogen/master.py", line 40, in <module>
import imp
ModuleNotFoundError: No module named 'imp'
Did not get where I am wrong? Patch is applied.
Tried with python3.12 (Fedora 38 made with virtual env)
If you are using a virtual env you will need to install the zombie-imp package.
On fedora it looks like that package is installed as part of the system so it works outside of the virtual env.
The description of the v2 patch says that you do not need it, but there still seems to be many 'import imp' statements, so that conversion doesn't seem to be complete? Anyway it is working very well for me with that install.
You should not need zombie-imp anymore. Please make sure to use the patch from this comment: #1033 (comment)
If that still does not work, yes, please install zombie-imp for a moment, until I fix it fully to not be needed.
but there still seems to be many 'import imp' statements
I see. There is still one in mitogen/master.py that needs to be addressed on the host side. And possibly the one in mitogen/compat/pkgutil.py
Will send v3 version today or on the weekend.
More issues when target is centos8(python3.6.8), host fedora39(3.12.1):
- import_tasks: redhat.yaml
when: ansible_os_family == 'RedHat'
ERROR! [mux 23009] 17:33:36.040315 E mitogen.[fork.2463684]: while importing 'ansible.utils.unsafe_proxy'
Traceback (most recent call last):
File "<stdin>", line 1666, in exec_module
File "master:/home/domas/.local/lib64/python3.12/site-packages/ansible/utils/unsafe_proxy.py", line 74
def __reduce__(self, /):
^
SyntaxError: invalid syntax
[mux 23009] 17:33:36.040857 D mitogen.io: Router(Broker(9f70))._async_route(Message(0, 1005, 1005, 102, 0, b'mitogen\x0040\x00unpickler.load exception\nTraceback (mos'..697), <Stream ssh.server.tld #2cf0>)
ERROR! [mux 23009] 17:33:36.041283 E mitogen.[fork.2463684]: unpickler.load exception
Traceback (most recent call last):
File "<stdin>", line 1001, in unpickle
File "<stdin>", line 787, in find_class
File "<stdin>", line 897, in _find_global
File "<stdin>", line 326, in lazy_AnsibleUnsafeText
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<stdin>", line 1666, in exec_module
File "master:/home/domas/.local/lib64/python3.12/site-packages/ansible/utils/unsafe_proxy.py", line 74
def __reduce__(self, /):
^
SyntaxError: invalid syntax
[mux 23009] 17:33:36.041734 D mitogen.io: Router(Broker(9f70))._async_route(Message(0, 1005, 1005, 102, 0, b"mitogen\x0040\x00raw pickle was: b'\\x80\\x02(NX\\x16\\x00\\x"..3308), <Stream ssh.server.tld #2cf0>)
ERROR! [mux 23009] 17:33:36.040315 E mitogen.[fork.2463684]: while importing 'ansible.utils.unsafe_proxy'
Traceback (most recent call last):
File "<stdin>", line 1666, in exec_module
File "master:/home/domas/.local/lib64/python3.12/site-packages/ansible/utils/unsafe_proxy.py", line 74
def __reduce__(self, /):
^
SyntaxError: invalid syntax
[mux 23009] 17:33:36.040857 D mitogen.io: Router(Broker(9f70))._async_route(Message(0, 1005, 1005, 102, 0, b'mitogen\x0040\x00unpickler.load exception\nTraceback (mos'..697), <Stream ssh.server.tld #2cf0>)
ERROR! [mux 23009] 17:33:36.041283 E mitogen.[fork.2463684]: unpickler.load exception
Traceback (most recent call last):
File "<stdin>", line 1001, in unpickle
File "<stdin>", line 787, in find_class
File "<stdin>", line 897, in _find_global
File "<stdin>", line 326, in lazy_AnsibleUnsafeText
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<stdin>", line 1666, in exec_module
File "master:/home/domas/.local/lib64/python3.12/site-packages/ansible/utils/unsafe_proxy.py", line 74
def __reduce__(self, /):
^
SyntaxError: invalid syntax
[mux 23009] 17:33:36.041734 D mitogen.io: Router(Broker(9f70))._async_route(Message(0, 1005, 1005, 102, 0, b"mitogen\x0040\x00raw pickle was: b'\\x80\\x02(NX\\x16\\x00\\x"..3308), <Stream ssh.server.tld #2cf0>)
- name: Install EPEL repository (Red Hat)
package:
name: epel-release
state: present
TASK [common : Install EPEL repository (Red Hat)] *************************************************************************************************************************************************************************************
task path: /home/domas/source/ansible-repo/roles/common/tasks/redhat.yaml:2
The full traceback is:
Traceback (most recent call last):
File "/home/domas/.local/lib64/python3.12/site-packages/ansible/executor/task_executor.py", line 165, in run
res = self._execute()
^^^^^^^^^^^^^^^
File "/home/domas/.local/lib64/python3.12/site-packages/ansible/executor/task_executor.py", line 641, in _execute
result = self._handler.run(task_vars=vars_copy)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/domas/.local/lib64/python3.12/site-packages/ansible_mitogen/mixins.py", line 146, in run
return super(ActionModuleMixin, self).run(tmp, task_vars)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/domas/.local/lib64/python3.12/site-packages/ansible/plugins/action/package.py", line 85, in run
result.update(self._execute_module(module_name=module, module_args=new_module_args, task_vars=task_vars, wrap_async=self._task.async_val))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/domas/.local/lib64/python3.12/site-packages/ansible_mitogen/mixins.py", line 386, in _execute_module
result = ansible_mitogen.planner.invoke(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/domas/.local/lib64/python3.12/site-packages/ansible_mitogen/planner.py", line 606, in invoke
response = _invoke_isolated_task(invocation, planner)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/domas/.local/lib64/python3.12/site-packages/ansible_mitogen/planner.py", line 519, in _invoke_isolated_task
return context.call(
^^^^^^^^^^^^^
File "/home/domas/.local/lib64/python3.12/site-packages/mitogen/parent.py", line 2023, in call
return self.default_call_chain.call(fn, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/domas/.local/lib64/python3.12/site-packages/mitogen/parent.py", line 1980, in call
return receiver.get().unpickle(throw_dead=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/domas/.local/lib64/python3.12/site-packages/mitogen/core.py", line 1013, in unpickle
raise obj
mitogen.core.CallError: builtins.SyntaxError: invalid syntax (unsafe_proxy.py, line 74)
File "<stdin>", line 3828, in _dispatch_one
File "<stdin>", line 3808, in _parse_request
File "<stdin>", line 1001, in unpickle
File "<stdin>", line 787, in find_class
File "<stdin>", line 897, in _find_global
File "<stdin>", line 326, in lazy_AnsibleUnsafeText
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<stdin>", line 1666, in exec_module
fatal: [server.tld]: FAILED! => {}
MSG:
Unexpected failure during module execution: builtins.SyntaxError: invalid syntax (unsafe_proxy.py, line 74)
File "<stdin>", line 3828, in _dispatch_one
File "<stdin>", line 3808, in _parse_request
File "<stdin>", line 1001, in unpickle
File "<stdin>", line 787, in find_class
File "<stdin>", line 897, in _find_global
File "<stdin>", line 326, in lazy_AnsibleUnsafeText
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<stdin>", line 1666, in exec_module
@kzinas-adv This was already mentioned in the very first comment I posted, second paragraph - #1033 (comment) I also posted there a workaround. There might be other way out, but it is rather complicated.
Workaround is to have python 3.8 on the target?
Workaround is to have python 3.8 on the target?
Either that, or manually patch ansible/utils/unsafe_proxy.py and remove all / references in argument lists in that file (this is a Python 3.8 feature).
I didn't finishe the v3 version of the patch, with the migogen/master.py ported fully to importlib. But it is shaping up.
@baryluk I've got some Fedora 39 systems now with python 3.12, so I'm eagerly awaiting your next version. Thank you very much for working on this!
Sorry for the long wait, but #1032 is ready for wider scrutiny. I'm seeking code review comments, and reports of anyone trying the branch in the wild moreati:docs-download-url. Note that on Python 3.12 it will require Ansible 6 (ansible-core 2.13). On Python 2.7 & 3.6-3.11 the supported Ansible versions are unchanged. @baryluk thanks for your work on this, would you like to be added to https://github.com/mitogen-hq/mitogen/blob/master/docs/contributors.rst?
@moreati Thanks for picking this up. I had few more minor updates (and removal of uneeded changes in my patch), but it was progressing slowly (because it kind of works, and I had other work to do, so didn't spend much on mitogen).
Your looks way better (including tests, and supporting legacy versions).
Yes, if you borrowed any code, please add me to the list, and we go with your changes.
Thanks!
Mitogen 0.3.5 released, with Python 3.12 support.