MagicStack/uvloop

Connection gets closed but connection_lost is not called

r00ta opened this issue · 1 comments

r00ta commented
  • uvloop version: 0.14
  • Python version: 3.8
  • Platform: Ubuntu 20.04
  • Can you reproduce the bug with PYTHONASYNCIODEBUG in env?: no
  • Does uvloop behave differently from vanilla asyncio? How?: Yes

We are using twisted with uvloop as event loop implementation.
On application side, we use the connection_lost and connection_made callback from asyncio.BaseProtocol to keep track of the connections with the clients.

Under heavy load, we've hit

File "/usr/lib/python3/dist-packages/twisted/internet/asyncioreactor.py", line 173, in addWriter
	self._asyncioEventloop.add_writer(fd, callWithLogger, writer,
File "uvloop/loop.pyx", line 2399, in uvloop.loop.Loop.add_writer
File "uvloop/loop.pyx", line 808, in uvloop.loop.Loop._add_writer
File "uvloop/handles/poll.pyx", line 122, in uvloop.loop.UVPoll.start_writing
File "uvloop/handles/poll.pyx", line 39, in uvloop.loop.UVPoll._poll_start
File "uvloop/handles/handle.pyx", line 159, in uvloop.loop.UVHandle._ensure_alive
builtins.RuntimeError: unable to perform operation on <UVPoll closed=True 0x7fe32f0f9cf0>; the handler is closed

and given that connection_lost was not called, we ended up in an infinite loop trying to reuse this closed connection.

I've investigated a bit the uvloop code and from my understanding libuv might raise silent exceptions that can close the poll without calling the connection_lost callback.

For example here

__on_uvpoll_event)

    cdef inline _poll_start(self, int flags):
        cdef int err
        print('checking alive')
        self._ensure_alive()

        err = uv.uv_poll_start(
            <uv.uv_poll_t*>self._handle,
            flags,
            __on_uvpoll_event)

and here

poll._fatal_error(exc, False)

cdef void __on_uvpoll_event(uv.uv_poll_t* handle,
                            int status, int events) with gil:

    if __ensure_handle_data(<uv.uv_handle_t*>handle, "UVPoll callback") == 0:
        return

    cdef:
        UVPoll poll = <UVPoll> handle.data

    if status < 0:
        exc = convert_error(status)
        poll._fatal_error(exc, False)
        return

as poll._fatal_error calls _close() directly.

I've managed to (kind of) reproduce this by patching uvloop, calling directly _fatal_error somewhere in the code and assessing that the connection_lost is not called.

Could you double check if my understanding is correct? I've being trying to force libuv to raise such exception, but I did not find a way to do it.

Looking at asyncio docs, connection_lost should be called also when the connection is closed https://docs.python.org/3/library/asyncio-protocol.html#asyncio.BaseProtocol.connection_lost

Thank you!

r00ta commented

Closing this as I've found that the issue was somewhere else.