Crash in metrics reporting
pipermerriam opened this issue · 0 comments
pipermerriam commented
What is wrong?
My node crashed today with this.
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: <Manager[BeamSyncService] flags=SRcfe>: task run[daemon=False] exited with error: [Errno -2] Name or service not known
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: Traceback (most recent call last):
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: File "/root/env/lib/python3.8/site-packages/async_service/base.py", line 324, in _run_and_manage_task
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: await task.run()
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: File "/root/env/lib/python3.8/site-packages/async_service/asyncio.py", line 35, in run
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: await self._async_fn(*self._async_fn_args)
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: File "/root/env/lib/python3.8/site-packages/trinity/sync/beam/service.py", line 60, in run
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: await self._pivot_loop()
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: File "/root/env/lib/python3.8/site-packages/trinity/sync/beam/service.py", line 80, in _pivot_loop
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: await self.sync_metrics_registry.record_pivot(latest_block)
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: File "/root/env/lib/python3.8/site-packages/trinity/components/builtin/metrics/sync_metrics_registry.py", line 24, in record_pivot
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: await self.metrics_service.send_annotation(post_data)
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: File "/root/env/lib/python3.8/site-packages/trinity/components/builtin/metrics/service/base.py", line 130, in send_annotation
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: await self._reporter.send_annotation(annotation_data)
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: File "/root/env/lib/python3.8/site-packages/trinity/components/builtin/metrics/service/base.py", line 43, in send_annotation
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: await self.session.post(annotation_data)
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: File "/root/env/lib/python3.8/site-packages/asks/sessions.py", line 206, in request
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: sock = await timeout_manager(
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: File "/root/env/lib/python3.8/site-packages/asks/utils.py", line 15, in timeout_manager
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: return await coro(*args)
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: File "/root/env/lib/python3.8/site-packages/asks/sessions.py", line 408, in _grab_connection
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: sock = await self._make_connection(host_loc)
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: File "/root/env/lib/python3.8/site-packages/asks/sessions.py", line 382, in _make_connection
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: sock, port = await self._connect(host_loc)
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: File "/root/env/lib/python3.8/site-packages/asks/sessions.py", line 105, in _connect
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: return await self._open_connection_https((host, int(port))), port
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: File "/root/env/lib/python3.8/site-packages/asks/sessions.py", line 76, in _open_connection_https
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: sock = await connect_tcp(
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: File "/root/env/lib/python3.8/site-packages/anyio/__init__.py", line 373, in connect_tcp
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: resolved = await getaddrinfo(target_host, port, family=family, type=socket.SOCK_STREAM)
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: File "/root/env/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 604, in getaddrinfo
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: result = await get_running_loop().getaddrinfo(
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: File "uvloop/loop.pyx", line 1469, in getaddrinfo
Sep 30 21:30:43 localhost launch-trinity.sh[17611]: socket.gaierror: [Errno -2] Name or service not known
How can it be fixed
🤷 probably simple exception handling from within the metrics service to gracefully handle this exception.