Dotaclient deadlocks on rmq connection loss
Closed this issue · 1 comments
TimZaman commented
Agent:
$ tzaman@Tims-Mac-Pro dotaclient (master) $ kubectl logs dotaservice-deployment-8c499c96-2mgbd agent
2019-01-06 21:18:41,153 INFO setup_model_cb(host=rmq.default.svc.cluster.local, port=5672)
2019-01-06 21:18:41,283 INFO Received new model: version=284, size=1207326b
2019-01-06 21:18:41,287 INFO Updated weights to version 284
2019-01-06 21:18:41,288 INFO === Starting Episode 0.
2019-01-06 21:18:41,288 INFO Starting game.
2019-01-06 21:18:41,290 ERROR error on dispatch
Traceback (most recent call last):
File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.7/asyncio/base_events.py", line 573, in run_until_complete
return future.result()
File "agent.py", line 515, in main
await game.play()
File "agent.py", line 430, in play
response = await asyncio.wait_for(self.dota_service.reset(self.config), timeout=120)
File "/usr/lib/python3.7/asyncio/tasks.py", line 416, in wait_for
return fut.result()
File "/root/.local/lib/python3.7/site-packages/grpclib/client.py", line 588, in __call__
await stream.send_message(message, end=True)
File "/root/.local/lib/python3.7/site-packages/grpclib/client.py", line 163, in send_message
await self.send_request()
File "/root/.local/lib/python3.7/site-packages/grpclib/client.py", line 132, in send_request
protocol = await self._channel.__connect__()
File "/root/.local/lib/python3.7/site-packages/grpclib/client.py", line 477, in __connect__
self._protocol = await self._create_connection()
File "/root/.local/lib/python3.7/site-packages/grpclib/client.py", line 465, in _create_connection
ssl=self._ssl)
File "/usr/lib/python3.7/asyncio/base_events.py", line 948, in create_connection
raise exceptions[0]
File "/usr/lib/python3.7/asyncio/base_events.py", line 935, in create_connection
await self.sock_connect(sock, address)
File "/usr/lib/python3.7/asyncio/selector_events.py", line 475, in sock_connect
return await fut
File "/usr/lib/python3.7/asyncio/selector_events.py", line 505, in _sock_connect_cb
raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 13337)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/.local/lib/python3.7/site-packages/aioamqp/protocol.py", line 333, in run
yield from self.dispatch_frame()
File "/root/.local/lib/python3.7/site-packages/aioamqp/protocol.py", line 288, in dispatch_frame
yield from channel.dispatch_frame(frame)
File "/root/.local/lib/python3.7/site-packages/aioamqp/channel.py", line 111, in dispatch_frame
yield from methods[(frame.class_id, frame.method_id)](frame)
File "/root/.local/lib/python3.7/site-packages/aioamqp/channel.py", line 631, in basic_deliver
content_body_frame = yield from self.protocol.get_frame()
File "/root/.local/lib/python3.7/site-packages/aioamqp/protocol.py", line 264, in get_frame
yield from frame.read_frame()
File "/root/.local/lib/python3.7/site-packages/aioamqp/frame.py", line 462, in read_frame
payload_data = yield from self.reader.readexactly(self.frame_length)
File "/usr/lib/python3.7/asyncio/streams.py", line 679, in readexactly
await self._wait_for_data('readexactly')
File "/usr/lib/python3.7/asyncio/streams.py", line 473, in _wait_for_data
await self._waiter
concurrent.futures._base.CancelledError
Dotaservice (0.3.2):
$ tzaman@Tims-Mac-Pro dotaclient (master) $ kubectl logs dotaservice-deployment-8c499c96-2mgbd dotaservice
2019-01-06 21:18:41,932 INFO DotaService 0.3.2 serving on :13337