3.11.5 regression: StreamWriter.del fails if event loop is already closed

Question

3.11.5 regression: StreamWriter.del fails if event loop is already closed

Closed this issue a year ago · 17 comments

Bug report

Bug description:

PR #107650 added a StreamWriter.__del__ that emits a ResourceWarning if a StreamWriter is not closed by the time it is garbage collected, and it has been backported as 3.11.5. However, if the event loop has already been closed by the time this happens, it causes a RuntimeError message to be displayed. It's non-fatal because exceptions raised by __del__ are ignored, but it causes an error message to be displayed to the user (as opposed to ResourceWarning, which is only shown when one opts in).

Code like the following used to run without any visible error, but now prints a traceback (and does not report the ResourceWarning for the unclosed StreamWriter when run with -X dev):

#!/usr/bin/env python3

import asyncio

async def main():
    global writer
    reader, writer = await asyncio.open_connection("127.0.0.1", 22)

asyncio.run(main())

Output in 3.11.5:

Exception ignored in: <function StreamWriter.__del__ at 0x7fd54ee11080>
Traceback (most recent call last):
  File "/usr/lib/python3.11/asyncio/streams.py", line 396, in __del__
  File "/usr/lib/python3.11/asyncio/streams.py", line 344, in close
  File "/usr/lib/python3.11/asyncio/selector_events.py", line 860, in close
  File "/usr/lib/python3.11/asyncio/base_events.py", line 761, in call_soon
  File "/usr/lib/python3.11/asyncio/base_events.py", line 519, in _check_closed
RuntimeError: Event loop is closed

CPython versions tested on:

3.11

Operating systems tested on:

Linux

Linked PRs

Answer 1 · 2023-09-27T15:36:42.000Z

Same problem here

Answer 2 · 2023-10-06T11:57:06.000Z

I have the same problem with 3.11.6. Thanks for reporting it.

Answer 3 · 2023-10-25T22:45:55.000Z

Is there a workaround?

Answer 4 · 2023-10-26T00:43:58.000Z

Is there a workaround?

Use gc.collect() may help

Here is my workaround.

@pytest.fixture
def event_loop():
    policy = asyncio.get_event_loop_policy()
    loop = policy.new_event_loop()
    loop.set_debug(True)
    yield loop
    gc.collect()
    loop.close()

Answer 5 · 2023-10-26T08:32:05.000Z

@dpr-0 Thanks, but just adding that to my program didn't prevent the messages.

Answer 6 · 2023-10-26T08:37:40.000Z

@dpr-0 Thanks, but just adding that to my program didn't prevent the messages.

Hope you can find the problem T^T

Answer 7 · 2023-11-09T16:44:22.000Z

Same problem for python 3.12.0

Answer 8 · 2023-11-09T18:00:31.000Z

Sorry for the inconvenience. Can you (or anyone else reading this) submit a PR for the main branch? We will then backport the PR to 3.12 and 3.11.

Answer 9 · 2023-11-10T01:57:51.000Z

Sorry for the inconvenience. Can you (or anyone else reading this) submit a PR for the main branch? We will then backport the PR to 3.12 and 3.11.

What is the desired behaviour if the event loop is already closed when the StreamWriter is destroyed? Should __del__ just become a no-op?

Answer 10 · 2023-11-10T04:03:06.000Z

What you expected -- I presume it'd still issue the resource warning, just not the "loop is closed" message.

Answer 11 · 2023-11-11T14:40:24.000Z

@gvanrossum I would imagine like this

def __del__(self, warnings=warnings):
    if not self._transport.is_closing():
        try:
            self.close()
        except RuntimeError:
            warnings.warn(f"loop is closed", ResourceWarning)
        else:
            warnings.warn(f"unclosed {self!r}", ResourceWarning)

If this one ok, I can submit a PR for this issues

Answer 12 · 2023-11-11T15:21:11.000Z

Sure, let’s give that a try as a PR.

Maybe separately it would be nice to gc.collect() in loop.close(). That’s a larger discussion though.

Answer 13 · 2023-11-22T17:25:02.000Z

Did anyone find a solution?

Answer 14 · 2023-11-22T18:52:45.000Z

Did anyone find a solution?

It's fixed in the main, 3.12 and 3.11 branches. It will be fixed in the next official releases of 3.11 and 3.12 from python.org (within 1-2 months).

Answer 15 · 2024-01-23T10:10:14.000Z

Did anyone find a solution?

It's fixed in the main, 3.12 and 3.11 branches. It will be fixed in the next official releases of 3.11 and 3.12 from python.org (within 1-2 months).

Is this fix included in 3.11.7?
Because I'm still able to reproduce it with Python 3.11.7

Edit: nevermind, I used 3.11.6 by accident. All good

Answer 16 · 2024-05-15T20:35:54.000Z

Just so you know, on python 3.11.9 with this setting for pytest:

[tool.pytest.ini_options]
asyncio_mode = "auto"
filterwarnings = ["error"]

the suite still raises an error from ignored warning:

    def _warn_teardown_exception(
        hook_name: str, hook_impl: HookImpl, e: BaseException
    ) -> None:
        msg = "A plugin raised an exception during an old-style hookwrapper teardown.\n"
        msg += f"Plugin: {hook_impl.plugin_name}, Hook: {hook_name}\n"
        msg += f"{type(e).__name__}: {e}\n"
        msg += "For more information see https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluggyTeardownRaisedWarning"  # noqa: E501
>       warnings.warn(PluggyTeardownRaisedWarning(msg), stacklevel=5)
E       pluggy.PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.
E       Plugin: unraisableexception, Hook: pytest_runtest_setup
E       PytestUnraisableExceptionWarning: Exception ignored in: <function StreamWriter.__del__ at 0xffffb9f4d620>
E       
E       Traceback (most recent call last):
E         File "/usr/local/lib/python3.11/asyncio/streams.py", line 411, in __del__
E           warnings.warn("loop is closed", ResourceWarning)
E       ResourceWarning: loop is closed
E       
E       For more information see https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluggyTeardownRaisedWarning

../virtualenvs/project-bTydf30B-py3.11/lib/python3.11/site-packages/pluggy/_callers.py:50: PluggyTeardownRaisedWarning

    def unraisable_exception_runtest_hook() -> Generator[None, None, None]:
        with catch_unraisable_exception() as cm:
            yield
            if cm.unraisable:
                if cm.unraisable.err_msg is not None:
                    err_msg = cm.unraisable.err_msg
                else:
                    err_msg = "Exception ignored in"
                msg = f"{err_msg}: {cm.unraisable.object!r}\n\n"
                msg += "".join(
                    traceback.format_exception(
                        cm.unraisable.exc_type,
                        cm.unraisable.exc_value,
                        cm.unraisable.exc_traceback,
                    )
                )
>               warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E               pytest.PytestUnraisableExceptionWarning: Exception ignored in: <function StreamWriter.__del__ at 0xffff878ed620>
E               
E               Traceback (most recent call last):
E                 File "/usr/local/lib/python3.11/asyncio/streams.py", line 411, in __del__
E                   warnings.warn("loop is closed", ResourceWarning)
E               ResourceWarning: loop is closed

../virtualenvs/project-bTydf30B-py3.11/lib/python3.11/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning

Is it expected? Should I add those warnings as ignored to my pytest setup?

Answer 17 · 2024-05-16T01:36:00.000Z

@mdczaplicki That error might be due to pytest -- I cannot tell from the output you're posting. If you really think this is still not fixed in CPython 3.11.9, can you open a new issue with complete instructions for reproducing the problem? (Ideally also testing in 3.12.)