Source hangs after ClientConnectorError and wont send data anymore
Closed this issue · 1 comments
kinnarr commented
Error log from deployment:
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: 2020-01-14 10:33:26,356 [ERROR ] [metricq.agent ] Future: <Task finished name='Task-37' coro=<IntervalSource.task() done, defined at /home/service/envs/source-rabbitmq/lib/python3.8/site-packages/metricq/interva
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: 2020-01-14 10:33:26,356 [CRITICAL] [metricq.agent ] Agent RabbitMqSource encountered an unhandled exception
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: Traceback (most recent call last):
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: File "/home/service/envs/source-rabbitmq/lib/python3.8/site-packages/aiohttp/connector.py", line 964, in _create_direct_connection
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: hosts = await asyncio.shield(self._resolve_host(
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: File "/home/service/envs/source-rabbitmq/lib/python3.8/site-packages/aiohttp/connector.py", line 829, in _resolve_host
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: addrs = await \
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: File "/home/service/envs/source-rabbitmq/lib/python3.8/site-packages/aiohttp/resolver.py", line 29, in resolve
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: infos = await self._loop.getaddrinfo(
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: File "/home/service/envs/source-rabbitmq/lib/python3.8/asyncio/base_events.py", line 817, in getaddrinfo
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: return await self.run_in_executor(
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: File "/home/service/envs/source-rabbitmq/lib/python3.8/concurrent/futures/thread.py", line 57, in run
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: result = self.fn(*self.args, **self.kwargs)
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: File "/home/service/envs/source-rabbitmq/lib/python3.8/socket.py", line 914, in getaddrinfo
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: socket.gaierror: [Errno -3] Temporary failure in name resolution
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: The above exception was the direct cause of the following exception:
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: Traceback (most recent call last):
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: File "/home/service/envs/source-rabbitmq/lib/python3.8/site-packages/metricq/interval_source.py", line 53, in task
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: await self.update()
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: File "/home/service/metricq-source-rabbitmq/metricq_source_rabbitmq/source.py", line 196, in update
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: await self._update_exchanges(session, vhost)
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: File "/home/service/metricq-source-rabbitmq/metricq_source_rabbitmq/source.py", line 87, in _update_exchanges
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: async with session.get(
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: File "/home/service/envs/source-rabbitmq/lib/python3.8/site-packages/aiohttp/client.py", line 1012, in __aenter__
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: self._resp = await self._coro
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: File "/home/service/envs/source-rabbitmq/lib/python3.8/site-packages/aiohttp/client.py", line 480, in _request
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: conn = await self._connector.connect(
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: File "/home/service/envs/source-rabbitmq/lib/python3.8/site-packages/aiohttp/connector.py", line 523, in connect
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: proto = await self._create_connection(req, traces, timeout)
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: File "/home/service/envs/source-rabbitmq/lib/python3.8/site-packages/aiohttp/connector.py", line 858, in _create_connection
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: _, proto = await self._create_direct_connection(
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: File "/home/service/envs/source-rabbitmq/lib/python3.8/site-packages/aiohttp/connector.py", line 971, in _create_direct_connection
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: raise ClientConnectorError(req.connection_key, exc) from exc
Jan 14 10:33:26 hase metricq-source-rabbitmq[70475]: aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host rabbitmq-server:443 ssl:default [Temporary failure in name resolution]