`RuntimeError` when the event loop is unset between tests
Closed this issue · 14 comments
When using a shared event loop (e.g., when loop_scope="session") and asyncio.set_event_loop(None) is called from a test, any subsequent async test raises RuntimeError: There is no current event loop in thread 'MainThread'.
This will happen in any test that calls asyncio.Runner or asyncio.run, which call asyncio.set_event_loop(None) as part of its clean-up code.
To reproduce:
import asyncio
import pytest
@pytest.mark.asyncio
async def test_before() -> None:
pass
def test_set_event_loop_none() -> None:
asyncio.set_event_loop(None)
@pytest.mark.asyncio
async def test_after() -> None:
pass$ pytest -o 'asyncio_default_test_loop_scope=module' test_mre.pyThe expected behaviour is for all tests to pass, but an exception is raised instead. The exception is not raised if either test_before or test_after are skipped.
I found issue #658 which is related; however, that issue is closed and related to fixtures (whereas this is not), so decided to create a new one instead.
I have a patch that resolves the issue by reinstating the event loop if needed. Let me know if a PR would be welcome!
Update: I bisected the issue to the v0.23 release, so this isn't a recent regression.
My understanding is that there are two cases where an event loop is unset:
- The user explicitly sets the loop to None
- The user calls third party code that sets the loop to None (e.g. because that third-party code doesn't play well with pytest-asyncio)
Can you share your use case and how you encountered the bug?
I currently think we should error out in both cases. The premise of using pytest-asyncio is to let the plugin manage the event loop. My worry is that things may break down the road if the user starts messing with the loop, even if we reinstate it.
Say the user has a long running fixture or tasks in the current loop and the loop gets replaced:
import asyncio
import pytest
import pytest_asyncio
@pytest_asyncio.fixture(scope="module", loop_scope="module")
async def webserver() -> None:
pass
@pytest.mark.asyncio(loop_scope="module")
async def test_before(webserver) -> None:
pass
def test_set_event_loop_none() -> None:
asyncio.set_event_loop(None)
@pytest.mark.asyncio(loop_scope="module")
async def test_after(webserver) -> None:
passCan we guarantee reinstating the module loop, as opposed to a fresh loop like the draft PR does?
If we can, I think we can give it a shot.
Can you share your use case and how you encountered the bug?
This was an issue I encountered when migrating a codebase from 0.21 to 1.1. It used to work on the old version whilst using the event_loop fixture that's no longer supported.
The situations in which this manifested were essentially
- code that schedules coroutines (such as on a background thread)
- code that apply a synchronous wrapper on asynchronous code (e.g., for code that doesn't support async directly)
- code that tests an entry point to an asynchronous program
All of which used calls to asyncio.Runner (and did not call asyncio.set_event_loop directly, to be clear). It was difficult to debug, as the test would still pass in isolation. The current solution (which I hope is temporary) was to reinstate the event loop in the code under test itself (using a helper function similar to the one in the draft PR), after the asyncio.Runner usage. Tests passed without issue once that was added.
I encountered this issue myself yesterday and worked around it by setting the test that exercised an entry point to use loop_scope="function". That seems to isolate it. Are you able to do that?
I encountered this issue myself yesterday and worked around it by setting the test that exercised an entry point to use loop_scope="function". That seems to isolate it. Are you able to do that?
This can happen due to regular non-async tests too as shown in the MRE, not sure it makes sense to set loop scope on sync tests.
From my experience encountering this issue in large codebase is that it becomes very difficult problem when you have hundreds of thousands of tests.
I actually added this annotation on a regular sync test and it worked. I
was surprised
Funny 😂
@seifertm I think @tjkuson PR makes sense to me at a glance. Would it be a reasonable path forward to make that behavior configurable under a setting e.g.
asyncio_reinstate_loopwhich defaults to false or would that not be a possible path forward @tjkuson?
@samypr100 I didn't want my comment to come across as dismissing the idea. Isolation between test cases is important for a lot of users and the reproducer shows that there's a gap. I merely wanted to point out an edge case (which should have been done as part of a PR review) :)
All of which used calls to
asyncio.Runner(and did not callasyncio.set_event_loopdirectly, to be clear). It was difficult to debug, as the test would still pass in isolation. The current solution (which I hope is temporary) was to reinstate the event loop in the code under test itself (using a helper function similar to the one in the draft PR), after theasyncio.Runnerusage. Tests passed without issue once that was added.
I haven't thought of this, but asyncio.Runner, and probably asyncio.run as well, set the current loop to None after closing it. It would greatly benefit test suites with interleaved sync/async test cases if we could address this.
With that in mind, we're aware of three occasions where the event loop can end up None:
- The user calls
set_event_loopexplicitly - A call to an
asynciofunction sets the loop - The user calls third party code that sets the loop to None
I'm currently experimenting with a solution that doesn't involve using get_event_loop and creating a fresh event loop.
The premise of using pytest-asyncio is to let the plugin manage the event loop.
I think reinstating the event-loop if it's unset is consistent with this principle. A user expecting the plugin to manage event loops would be surprised that a synchronous test (that otherwise appears entirely isolated and unmanaged by pytest-asyncio) could unset an event loop managed by pytest-asyncio causing further test execution to error.
I'm currently experimenting with a solution that doesn't involve using
get_event_loopand creating a fresh event loop.
I've got a local patch working that stores the managed event loops and restores the appropriate one if unset (satisfying the example you provided). It's a more invasive change than the current draft PR and is currently quite unfinished; is that more what you were thinking?
I've spent 2 days struggling with this, I had instances of this problem with sqlalchemy[asyncpg] and with valkey running async implementation too.
I was forced to rollback to 0.26.0, here is my attempt at troubleshooting it that I posted on valkey github:
I've tried everything, from changing scopes, changing types of implementations, nothing hit the nail on the head, downgrading pytest-asyncio to 0.26.0 was an instant fix
I encountered the similar issue with v1.1.0. Downgrading to v1.0.0 resolved the error for now.
import asyncio
import pytest
async def do_work():
await asyncio.sleep(0.05)
return 42
@pytest.mark.asyncio(loop_scope="module")
async def test_async_helper_function():
result = await do_work()
assert result == 42
With v1.1.0, I have the error RuntimeError: There is no current event loop in thread 'MainThread'
================================================= test session starts ==================================================
platform linux -- Python 3.12.10, pytest-8.4.1, pluggy-1.6.0
rootdir: src/pytest_async
configfile: pytest.ini
plugins: asyncio-1.1.0
asyncio: mode=Mode.STRICT, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 1 item
tests/test_sleep.py .E [100%]
======================================================== ERRORS ========================================================
___________________________________ ERROR at teardown of test_async_helper_function ____________________________________
event_loop_policy = <asyncio.unix_events._UnixDefaultEventLoopPolicy object at 0x7ebf84f87b30>
@pytest.fixture(
scope=scope,
name=f"_{scope}_scoped_runner",
)
def _scoped_runner(
event_loop_policy,
) -> Iterator[Runner]:
new_loop_policy = event_loop_policy
> with _temporary_event_loop_policy(new_loop_policy):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.12/site-packages/pytest_asyncio/plugin.py:756:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/contextlib.py:144: in __exit__
next(self.gen)
.venv/lib/python3.12/site-packages/pytest_asyncio/plugin.py:546: in _temporary_event_loop_policy
_set_event_loop(old_loop)
.venv/lib/python3.12/site-packages/pytest_asyncio/plugin.py:575: in _set_event_loop
asyncio.set_event_loop(loop)
../../.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/asyncio/events.py:817: in set_event_loop
l_before = get_event_loop()
^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <asyncio.unix_events._UnixDefaultEventLoopPolicy object at 0x7ebf84f87b30>
def get_event_loop(self):
"""Get the event loop for the current context.
Returns an instance of EventLoop or raises an exception.
"""
if (self._local._loop is None and
not self._local._set_called and
threading.current_thread() is threading.main_thread()):
stacklevel = 2
try:
f = sys._getframe(1)
except AttributeError:
pass
else:
# Move up the call stack so that the warning is attached
# to the line outside asyncio itself.
while f:
module = f.f_globals.get('__name__')
if not (module == 'asyncio' or module.startswith('asyncio.')):
break
f = f.f_back
stacklevel += 1
import warnings
warnings.warn('There is no current event loop',
DeprecationWarning, stacklevel=stacklevel)
self.set_event_loop(self.new_event_loop())
if self._local._loop is None:
> raise RuntimeError('There is no current event loop in thread %r.'
% threading.current_thread().name)
E RuntimeError: There is no current event loop in thread 'MainThread'.
../../.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/asyncio/events.py:702: RuntimeError
------------------------------------------------ Captured stdout setup -------------------------------------------------
set_event_loop: <_UnixSelectorEventLoop running=False closed=False debug=False> -> <_UnixSelectorEventLoop running=False closed=False debug=False>
=============================================== short test summary info ================================================
ERROR tests/test_sleep.py::test_async_helper_function - RuntimeError: There is no current event loop in thread 'MainThread'.
@DragonRollDev Thanks for reporting that a rollback to an earlier version helped. The issue in and of itself can be hard to debug and I can imagine it gets even harder when sqlalchemy is involved (due to the traces produced by greenlet + asyncio use of async sqlalchemy).
@dunalduck0 I think your reproducer is incomplete. Your error message shows that one test succeeded, but the other failed. However, the reproducer contains only a single test. In any case, the rollback information is helpful, thank you!
It’s actually 1 succeeded and 1 error. The single test succeeded but an error was thrown at teardown time (in the code of purest.mark.asyncio wrapper).
This issue was addressed in pytest-asyncio v.1.2.0.