Threading concurrency problem: dictionary changed during iteration

[pipermerriam]

What is wrong?

This crashed one of my nodes.

Sep 30 11:22:33 localhost launch-trinity.sh[10995]:     raise MultiError(
Sep 30 11:22:33 localhost launch-trinity.sh[10995]:   File "/root/env/lib/python3.8/site-packages/async_service/base.py", line 324, in _run_and_manage_task
Sep 30 11:22:33 localhost launch-trinity.sh[10995]:     await task.run()
Sep 30 11:22:33 localhost launch-trinity.sh[10995]:   File "/root/env/lib/python3.8/site-packages/async_service/base.py", line 169, in run
Sep 30 11:22:33 localhost launch-trinity.sh[10995]:     await self.child_manager.run()
Sep 30 11:22:33 localhost launch-trinity.sh[10995]:   File "/root/env/lib/python3.8/site-packages/async_service/asyncio.py", line 255, in run
Sep 30 11:22:33 localhost launch-trinity.sh[10995]:     raise MultiError(
Sep 30 11:22:33 localhost launch-trinity.sh[10995]:   File "/root/env/lib/python3.8/site-packages/async_service/base.py", line 324, in _run_and_manage_task
Sep 30 11:22:33 localhost launch-trinity.sh[10995]:     await task.run()
Sep 30 11:22:33 localhost launch-trinity.sh[10995]:   File "/root/env/lib/python3.8/site-packages/async_service/base.py", line 169, in run
Sep 30 11:22:33 localhost launch-trinity.sh[10995]:     await self.child_manager.run()
Sep 30 11:22:33 localhost launch-trinity.sh[10995]:   File "/root/env/lib/python3.8/site-packages/async_service/asyncio.py", line 255, in run
Sep 30 11:22:33 localhost launch-trinity.sh[10995]:     raise MultiError(
Sep 30 11:22:33 localhost launch-trinity.sh[10995]:   File "/root/env/lib/python3.8/site-packages/async_service/base.py", line 324, in _run_and_manage_task
Sep 30 11:22:33 localhost launch-trinity.sh[10995]:     await task.run()
Sep 30 11:22:33 localhost launch-trinity.sh[10995]:   File "/root/env/lib/python3.8/site-packages/async_service/asyncio.py", line 35, in run
Sep 30 11:22:33 localhost launch-trinity.sh[10995]:     await self._async_fn(*self._async_fn_args)
Sep 30 11:22:33 localhost launch-trinity.sh[10995]:   File "/root/env/lib/python3.8/site-packages/trinity/sync/beam/state.py", line 567, in _match_predictive_node_requests_to_peers
Sep 30 11:22:33 localhost launch-trinity.sh[10995]:     await self._wakeup_preview_waiters(existing_hashes)
Sep 30 11:22:33 localhost launch-trinity.sh[10995]:   File "/root/env/lib/python3.8/site-packages/trinity/sync/beam/state.py", line 671, in _wakeup_preview_waiters
Sep 30 11:22:33 localhost launch-trinity.sh[10995]:     preview_waiters = await asyncio.get_event_loop().run_in_executor(
Sep 30 11:22:33 localhost launch-trinity.sh[10995]:   File "/root/.pyenv/versions/3.8.5/lib/python3.8/concurrent/futures/thread.py", line 57, in run
Sep 30 11:22:33 localhost launch-trinity.sh[10995]:     result = self.fn(*self.args, **self.kwargs)
Sep 30 11:22:33 localhost launch-trinity.sh[10995]:   File "/root/env/lib/python3.8/site-packages/trinity/sync/beam/state.py", line 684, in _get_preview_waiters
Sep 30 11:22:33 localhost launch-trinity.sh[10995]:     waiters = tuple(self._preview_events.items())
Sep 30 11:22:33 localhost launch-trinity.sh[10995]: RuntimeError: dictionary changed size during iteration

How can it be fixed

Looks like we need some concurrency protectins/mitigation now that we're running more things in threads.

[pipermerriam]

Hit this one again today.