Task got Future attached to a different loop error while migrating from aioredis 1.3.1 to 2.0

[kiranpradeep]

I was trying migrate redis-py application to aioredis 2.0. It used redis client in callbacks. Below code sample is minimal example which works on aioredis 1.3.1, but fails on aioredis 2.0. It reports RuntimeError: Task got Future <Future pending> attached to a different loop. How should we use the redis client as class variable when there are callbacks ?

import aioredis
import asyncio

class Main:
    async def create_client(self):
        self.client = aioredis.Redis.from_url("redis://localhost:6379") # fails for aioredis 2.0
        # self.client = await aioredis.create_redis_pool("redis://localhost:6379") # works for aioredis 1.3.1

    async def my_fun(self):
        await self.client.set("key1", "value")

    async def my_callback(self):
        await self.client.set("key2", "value") # throws error here

main = Main()
asyncio.run(main.create_client())
asyncio.run(main.my_fun())
asyncio.run(main.my_callback()) # consider this callback from a different loop like mqtt subscriber message received

Running above program for aioredis 2.0 gives below error. I have earlier raised similar issue #941, but raising this to understand what the recommended solution is in these cases. To give context, my callback happens from a different library (mqtt subscriber message) which runs its own loop .

Traceback (most recent call last):
File "/home/roots/miniconda3/envs/iotedge/lib/python3.9/site-packages/aioredis/connection.py", line 852, in read_response
response = await self._parser.read_response()
File "/home/roots/miniconda3/envs/iotedge/lib/python3.9/site-packages/aioredis/connection.py", line 368, in read_response
raw = await self._buffer.readline()
File "/home/roots/miniconda3/envs/iotedge/lib/python3.9/site-packages/aioredis/connection.py", line 302, in readline
await self._read_from_socket()
File "/home/roots/miniconda3/envs/iotedge/lib/python3.9/site-packages/aioredis/connection.py", line 248, in _read_from_socket
data = await self._stream.read(self.socket_read_size)
File "/home/roots/miniconda3/envs/iotedge/lib/python3.9/asyncio/streams.py", line 684, in read
await self._wait_for_data('read')
File "/home/roots/miniconda3/envs/iotedge/lib/python3.9/asyncio/streams.py", line 517, in _wait_for_data
await self._waiter
RuntimeError: Task <Task pending name='Task-7' coro=<Main.my_callback() running at /mnt/d/kiran/src/git/elz/IoTEdge/modules/DetectionTracer/app/test2.py:14> cb=[_run_until_complete_cb() at /home/roots/miniconda3/envs/iotedge/lib/python3.9/asyncio/base_events.py:184]> got Future attached to a different loop

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/mnt/d/kiran/src/git/elz/IoTEdge/modules/DetectionTracer/app/test2.py", line 20, in
asyncio.run(main.my_callback())
File "/home/roots/miniconda3/envs/iotedge/lib/python3.9/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/home/roots/miniconda3/envs/iotedge/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
File "/mnt/d/kiran/src/git/elz/IoTEdge/modules/DetectionTracer/app/test2.py", line 14, in my_callback
await self.client.set("key2", "value")
File "/home/roots/miniconda3/envs/iotedge/lib/python3.9/site-packages/aioredis/client.py", line 1065, in execute_command
return await self.parse_response(conn, command_name, **options)
File "/home/roots/miniconda3/envs/iotedge/lib/python3.9/site-packages/aioredis/client.py", line 1081, in parse_response
response = await connection.read_response()
File "/home/roots/miniconda3/envs/iotedge/lib/python3.9/site-packages/aioredis/connection.py", line 857, in read_response
await self.disconnect()
File "/home/roots/miniconda3/envs/iotedge/lib/python3.9/site-packages/aioredis/connection.py", line 760, in disconnect
self._writer.close()
File "/home/roots/miniconda3/envs/iotedge/lib/python3.9/asyncio/streams.py", line 353, in close
return self._transport.close()
File "/home/roots/miniconda3/envs/iotedge/lib/python3.9/asyncio/selector_events.py", line 700, in close
self._loop.call_soon(self._call_connection_lost, None)
File "/home/roots/miniconda3/envs/iotedge/lib/python3.9/asyncio/base_events.py", line 746, in call_soon
self._check_closed()
File "/home/roots/miniconda3/envs/iotedge/lib/python3.9/asyncio/base_events.py", line 510, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed