nest asyncio causing deadlock?

[dpatel20]

I have the following piece of code that uses python-can and aioisotp (tried to simplify it). Although this particular code has no need for nest_asyncio my larger code uses it and so I need this to work with nest_asyncio.

import nest_asyncio
nest_asyncio.apply() 

import time
import binascii
import can
from can.interfaces.vector import VectorBus
import asyncio
import aioisotp

async def infinite(rdr):
    while True:
        payload = await rdr.read(4095)
        print("inf_rdr:", binascii.hexlify(payload).decode())
        
async def main(): 
    network = aioisotp.ISOTPNetwork(0,
                                    interface='vector',
                                    receive_own_messages=False, tx_padding=0x0)
    with network.open():
        reader, writer = await network.open_connection(0x7BB, 0x7B3)
        reader1, writer1 = await network.open_connection(0x728, 0x720)
        reader2, writer2 = await network.open_connection(0x7DF, 0x7DF)
        asyncio.create_task(infinite(reader))
        asyncio.create_task(infinite(reader1))

        writer2.write(b'\x22\xF1\x13')

        await asyncio.sleep(6.5) # The read() in infinite() does not complete until this times out
    await asyncio.sleep(0.5)
    
if __name__ == "__main__":
    asyncio.run(main())

What I find, is the write() occurs quickly as expected. However, (per the comment in the code) the read() in infinite() seems to be blocked until await asyncio.sleep(6.5) times out - so the print() occurs after 6.5s. Whatever time I change that sleep() to, that's how long before the print() is seen. If I remove the nest_asyncio, it works as expected - i.e. the read() completes as soon as the data arrives (milliseconds).

Perhaps I am misunderstanding what nest_asyncio does but it seems with it, it 'blocks' the read?

I have successfully used nest_asyncio in other code but I can't see if I am using it incorrectly here. Any ideas if this is my code/understanding or a bug?

(Python 3.8)

[erdewit]

If I run the example on Linux it fails with ImportError: The Vector API has not been loaded. Not sure what dependencies are needed but it looks like they're Windows-specific.

Perhaps try the SelectorEventLoop instead of proactor.

Do you use the latest version of nest_asyncio?

[dpatel20]

Yep, Vector is specific interface only available on Windows.

I am on latest (v1.4.3).

With regards to this:

Perhaps try the SelectorEventLoop instead of proactor.

I tried the following and it seems to work (at least for the above code):

if __name__ == "__main__":
   selector = selectors.SelectSelector()
   loop = asyncio.SelectorEventLoop(selector)
   asyncio.set_event_loop(loop)
   import nest_asyncio
   nest_asyncio.apply(loop)
   loop.run_until_complete(main())

Is this what you meant? This seems to fix the issue for the code above but I will need to test more on the other code to ensure the part that needs the re-entrant asyncio still works.

Can you explain what this change does exactly?

[dpatel20]

Did some testing and I think using SelectorEventLoop works even in the nested asyncio loop cases.
But I would like to understand if nest_asyncio should work in this case using the Windows ProactorEventLoop?

[erdewit]

It should work with the ProactorEventLoop as well, but that loop is much more complex and I have little experience with the implementation.

The example has a dependency on running some special hardware interface, making it impossible for me to reproduce the problem. If you have a test case that runs as-is and shows a hanging ProactorEventLoop that would be appreciated. Otherwise it will not really be possible to solve this.

[dpatel20]

I think the following should show the issue on Windows without need of any hardware. As is, the infinite() will print. If you change useProactor = False to True you never get any print out even after the sleep (so slightly different to what I saw with real hardware but hopefully same root cause).

The project I am using for this is https://github.com/christiansandberg/aioisotp. Not sure if it is something to do with how that is implemented as I only see this issue when using that library.

import asyncio
import aioisotp
import selectors

async def infinite(rdr):
    while True:
        payload = await rdr.read(4095)
        print("inf_rdr:", payload)

class EchoServer(asyncio.Protocol):

    def connection_made(self, transport):
        self.transport = transport

    def data_received(self, data):
        # Echo back the same data
        self.transport.write(data)


async def main():
    network = aioisotp.ISOTPNetwork('vcan0',
                                    interface='virtual',
                                    receive_own_messages=True)
    with network.open():
        # A server that uses a protocol
        transport, protocol = await network.create_connection(
            EchoServer, 0x1CDADCF9, 0x1CDAF9DC)

        # A client that uses streams
        reader, writer = await network.open_connection(
            0x1CDAF9DC, 0x1CDADCF9)
        asyncio.create_task(infinite(reader))
        writer.write(b'Hello world!')
        await asyncio.sleep(6.5)

useProactor = False
if useProactor:
    loop = asyncio.get_event_loop()
else:
    selector = selectors.SelectSelector()
    loop = asyncio.SelectorEventLoop(selector)
    asyncio.set_event_loop(loop)
import nest_asyncio
nest_asyncio.apply(loop)
loop.run_until_complete(main())

[erdewit]

Thank you very much for the updated test case. I can reproduce the issue and know now how to fix it.

[erdewit]

The fix is released in v1.5.0.

[dpatel20]

Thanks. Tested the fix and it seems to working nicely!