Genshi doesn't work on Python 3.11: code expected at least 18 arguments, got 16

Question

Genshi doesn't work on Python 3.11: code expected at least 18 arguments, got 16

vstinner opened this issue 3 years ago · 14 comments

Example of error when running tests:

======================================================================
FAIL: OldTextTemplate (genshi.template.text)
Doctest: genshi.template.text.OldTextTemplate
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/doctest.py", line 2216, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for genshi.template.text.OldTextTemplate
  File "/home/vstinner/dev/genshi/genshi/template/text.py", line 239, in OldTextTemplate

----------------------------------------------------------------------
File "/home/vstinner/dev/genshi/genshi/template/text.py", line 244, in genshi.template.text.OldTextTemplate
Failed example:
    tmpl = OldTextTemplate('''Dear $name,

    We have the following items for you:
    #for item in items
     * $item
    #end

    All the best,
    Foobar''')
Exception raised:
    Traceback (most recent call last):
      File "/home/vstinner/python/main/Lib/doctest.py", line 1348, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest genshi.template.text.OldTextTemplate[0]>", line 1, in <module>
        tmpl = OldTextTemplate('''Dear $name,
      File "/home/vstinner/dev/genshi/genshi/template/base.py", line 421, in __init__
        self._stream = self._parse(source, encoding)
      File "/home/vstinner/dev/genshi/genshi/template/text.py", line 293, in _parse
        for kind, data, pos in interpolate(text, self.filepath, lineno,
      File "/home/vstinner/dev/genshi/genshi/template/interpolation.py", line 77, in interpolate
        expr = Expression(chunk.strip(), pos[0], pos[1],
      File "/home/vstinner/dev/genshi/genshi/template/eval.py", line 71, in __init__
        self.code = _compile(node, self.source, mode=self.mode,
      File "/home/vstinner/dev/genshi/genshi/template/eval.py", line 453, in _compile
        return build_code_chunk(code, filename, name, lineno)
      File "/home/vstinner/dev/genshi/genshi/compat.py", line 110, in build_code_chunk
        return CodeType(*params)
    TypeError: code() argument 15 must be bytes, not tuple
----------------------------------------------------------------------
File "/home/vstinner/dev/genshi/genshi/template/text.py", line 253, in genshi.template.text.OldTextTemplate
Failed example:
    print(tmpl.generate(name='Joe', items=[1, 2, 3]).render(encoding=None))
Exception raised:
    Traceback (most recent call last):
      File "/home/vstinner/python/main/Lib/doctest.py", line 1348, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest genshi.template.text.OldTextTemplate[1]>", line 1, in <module>
        print(tmpl.generate(name='Joe', items=[1, 2, 3]).render(encoding=None))
    NameError: name 'tmpl' is not defined

Answer 1 · 2021-05-20T21:31:02.000Z

While you're waiting for official docs, here's the new constructor:
https://github.com/python/cpython/blob/18f41c04ff4161531f4d08631059fd3ed37c0218/Lib/test/test_code.py#L216-L232

Answer 2 · 2021-09-17T11:00:16.000Z

@vstinner Would you mind testing this again? In a23f305 I changed the code so we don't use the constructor anymore and maybe that also fixed the 3.11 problem. (Fedora has now 3.11 package yet so testing this is a bit more complicated for me.)

Answer 3 · 2021-09-17T12:04:17.000Z

@FelixSchwarz: I still get test errors, in another function:

======================================================================
ERROR: test_pickle (genshi.template.tests.eval.ExpressionTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/dev/genshi/genshi/template/tests/eval.py", line 45, in test_pickle
    unpickled = pickle.load(buf)
                ^^^^^^^^^^^^^^^^
  File "/home/vstinner/dev/genshi/genshi/template/eval.py", line 92, in __setstate__
    self.code = CodeType(0, *state['code'])
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: code expected at least 18 arguments, got 16

======================================================================
ERROR: test_pickle (genshi.template.tests.eval.SuiteTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/dev/genshi/genshi/template/tests/eval.py", line 551, in test_pickle
    unpickled = pickle.load(buf)
                ^^^^^^^^^^^^^^^^
  File "/home/vstinner/dev/genshi/genshi/template/eval.py", line 92, in __setstate__
    self.code = CodeType(0, *state['code'])
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: code expected at least 18 arguments, got 16

======================================================================
ERROR: test_pickle (genshi.template.tests.markup.MarkupTemplateTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/dev/genshi/genshi/template/tests/markup.py", line 52, in test_pickle
    unpickled = pickle.load(buf)
                ^^^^^^^^^^^^^^^^
  File "/home/vstinner/dev/genshi/genshi/template/eval.py", line 92, in __setstate__
    self.code = CodeType(0, *state['code'])
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: code expected at least 18 arguments, got 16

Commands:

$ ~/python/main/python -m venv env  # Python 3.11 (dev)
$ env/bin/python setup.py test

Maybe Code.__setstate__() should recompile the code using self.code = _compile(...) rather than serializing CodeType parameters.

Answer 4 · 2021-09-17T12:06:30.000Z

To build Python 3.11:

git clone https://github.com/python/cpython/
cd cpython
./configure
make

No need to install it: just use the ./python build in the root directory of the source code.

Answer 5 · 2021-09-17T12:09:22.000Z

Ah, that is the part I left open in #49 but @hodgestar is currently working on some fixes as part of #50 (for Python 3.10). Yeah, guess I'll have to set up a Python 3.11 then (though free time is pretty limited - as always :-).

Thank you for re-testing this.

Answer 6 · 2021-10-31T23:46:21.000Z

Since I was doing this for my own templating engine Tonnikala and was just casually browsing around, I noticed that Genshi seems to have accidentally swapped co_kwonlyargcount and co_nlocals c.f. CPython help() for code objects...

Answer 7 · 2021-11-01T07:54:36.000Z

@ztane thank you for reporting this. I filed #52 so this won't get lost.

Answer 8 · 2021-11-13T09:21:09.000Z

@vstinner Thank you for your comment how to build Python 3.11 locally. It worked and I created PR #54 as a first step to make Genshi compatible with Python 3.11.

Answer 9 · 2022-01-06T15:27:03.000Z

You cannot use CodeType.replace() to avoid being broken each time CodeType constructor is changed?

Answer 10 · 2022-01-06T15:47:23.000Z

@vstinner We do use CodeType.replace for Python 3.8 in build_code_chunk, but get_code_params is used to pickle and unpickle the code object, so I don't know if CodeType.replace can help in that case.

Answer 11 · 2022-01-06T20:57:14.000Z

Ah right, pickling code objects is non-trivial.

Answer 12 · 2022-01-06T21:39:54.000Z

@vstinner Do you think CPython would be interested in support for serializing or pickling code objects? That would mean that perhaps one day this continual dance of fixing this code could end.

Answer 13 · 2022-01-06T21:59:54.000Z

Hm, could you marshal the code object instead, and use those bytes to "unmarshal" later? Not sure if that would be appropriate for your use-case.

Answer 14 · 2022-01-07T07:46:04.000Z

Do you think CPython would be interested in support for serializing or pickling code objects?

I don't know. You can propose the idea on python-dev or python-ideas.