Electrumx crash and cannot recover after detecting a reorg
jplix7k opened this issue · 2 comments
Encountered twice on BCH testnet recently, ver 1.14 is used. Need help!
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:electrumx:ElectrumX server starting
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:electrumx:logging level: INFO
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:Controller:Python version: 3.7.5 (default, Nov 7 2019, 10:50:52) [GCC 8.3.0]
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:Controller:software version: ElectrumX 1.14.0
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:Controller:aiorpcX version: 0.18.4
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:Controller:supported protocol versions: 1.4-1.4.2
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:Controller:event loop policy: None
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:Controller:reorg limit is 8,000 blocks
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:Daemon:daemon #1 at 10.3.64.175:18332/ (current)
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:DB:switching current directory to /work/electrumx_db2
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:DB:using leveldb for DB backend
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:DB:opened UTXO DB (for sync: True)
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:DB:DB version: 7
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:DB:coin: BitcoinCashABC
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:DB:network: testnet
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:DB:height: 1,383,453
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:DB:tip: 0000000005b16c3d56ebfec3116e8b8258046d31c39f6769a856165683bdad0b
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:DB:tx count: 63,578,607
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:DB:flushing DB cache at 1,200 MB
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:History:history DB version: 0
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:History:flush count: 527
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:Prefetcher:catching up to daemon height 1,383,649 (196 blocks behind)
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:SessionManager:RPC server listening on localhost:8000
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:LTORBlockProcessor:chain reorg detected
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:LTORBlockProcessor:chain was reorganised replacing 1 block at heights 1,383,453-1,383,453
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:LTORBlockProcessor:read 1 blocks from disk
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:Prefetcher:cancelled; prefetcher stopping
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:LTORBlockProcessor:flushing to DB for a clean shutdown...
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:SessionManager:closing down server for rpc://localhost:8000
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:Controller:shutting down
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: INFO:Controller:shutdown complete
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: ERROR:electrumx:ElectrumX server terminated abnormally
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: Traceback (most recent call last):
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: File "/usr/local/lib/python3.7/dist-packages/electrumX-1.14.0-py3.7.egg/EGG-INFO/scripts/electrumx_server", line 35, in main
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: return loop.run_until_complete(main)
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: File "/usr/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: return future.result()
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: File "/usr/local/lib/python3.7/dist-packages/electrumX-1.14.0-py3.7.egg/electrumx/lib/server_base.py", line 125, in run
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: await server_task
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: File "/usr/local/lib/python3.7/dist-packages/electrumX-1.14.0-py3.7.egg/electrumx/lib/server_base.py", line 98, in serve
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: await self.serve(shutdown_event)
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: File "/usr/local/lib/python3.7/dist-packages/electrumX-1.14.0-py3.7.egg/electrumx/server/controller.py", line 134, in serve
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: await group.spawn(wait_for_catchup())
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: File "/usr/local/lib/python3.7/dist-packages/aiorpcX-0.18.4-py3.7.egg/aiorpcx/curio.py", line 242, in aexit
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: await self.join()
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: File "/usr/local/lib/python3.7/dist-packages/aiorpcX-0.18.4-py3.7.egg/aiorpcx/curio.py", line 211, in join
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: raise task.exception()
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: File "/usr/local/lib/python3.7/dist-packages/electrumX-1.14.0-py3.7.egg/electrumx/server/block_processor.py", line 681, in fetch_and_process_blocks
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: await self.flush(True)
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: File "/usr/local/lib/python3.7/dist-packages/electrumX-1.14.0-py3.7.egg/electrumx/server/block_processor.py", line 343, in flush
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: await self.run_in_thread_with_lock(flush)
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: File "/usr/local/lib/python3.7/dist-packages/electrumX-1.14.0-py3.7.egg/electrumx/server/block_processor.py", line 199, in run_in_thread_with_lock
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: return await asyncio.shield(run_in_thread_locked())
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: File "/usr/local/lib/python3.7/dist-packages/electrumX-1.14.0-py3.7.egg/electrumx/server/block_processor.py", line 198, in run_in_thread_locked
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: return await run_in_thread(func, *args)
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: File "/usr/local/lib/python3.7/dist-packages/aiorpcX-0.18.4-py3.7.egg/aiorpcx/curio.py", line 68, in run_in_thread
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: return await get_event_loop().run_in_executor(None, func, *args)
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: result = self.fn(*self.args, **self.kwargs)
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: File "/usr/local/lib/python3.7/dist-packages/electrumX-1.14.0-py3.7.egg/electrumx/server/block_processor.py", line 342, in flush
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: self.estimate_txs_remaining)
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: File "/usr/local/lib/python3.7/dist-packages/electrumX-1.14.0-py3.7.egg/electrumx/server/db.py", line 202, in flush_dbs
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: self.assert_flushed(flush_data)
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: File "/usr/local/lib/python3.7/dist-packages/electrumX-1.14.0-py3.7.egg/electrumx/server/db.py", line 190, in assert_flushed
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: assert flush_data.tip == self.db_tip
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: AssertionError
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: /usr/local/lib/python3.7/dist-packages/aiohttp-4.0.0a1-py3.7-linux-x86_64.egg/aiohttp/client.py:977: RuntimeWarning: coroutine 'noop' was never awaited
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: self._resp.release()
Jun 04 07:32:03 pool1-sl-bch electrumx_server[9068]: RuntimeWarning: Enable tracemalloc to get the object allocation traceback
packet_write_wait: Connection to 3.112.251.189 port 22: Broken pipe
It happened again. Can you please give some help? Thank you.
I'm sorry Bitcoin Cash is no longer supported; I think their developers maintain their own repo or have different software