Error when loading last N blocks using JSONRPCReorganisationMonitor

[bryaan]

How do you only load the last N blocks? This code wants to loads only last 5 blocks i believe, however it errors when adding new blocks.

    reorg_mon = JSONRPCReorganisationMonitor(web3, check_depth=30)

    reorg_mon.load_initial_block_headers(block_count=5)

    while True:
        try:
            # Figure out the next good unscanned block range,
            # and fetch block headers and timestamps for this block range
            chain_reorg_resolution = reorg_mon.update_chain()

            if chain_reorg_resolution.reorg_detected:
                logger.info(f"Chain reorganisation data updated: {chain_reorg_resolution}")

            # Read specified events in block range
            for log_result in read_events(
                web3,
                start_block=chain_reorg_resolution.latest_block_with_good_data + 1,
                end_block=chain_reorg_resolution.last_live_block,
                filter=filter,
                notify=None,
                chunk_size=100,
                context=token_cache,
                extract_timestamps=None,
                reorg_mon=reorg_mon,
            ):
               pass

INFO:eth_defi.event_reader.reorganisation_monitor:figure_reorganisation_and_new_blocks(), range 17,285,423 - 17,285,443, last block we have is 17,285,443, check depth is 20
ERROR: LoadError: Python: AssertionError: Blocks must be added in order. Last block we have: 17285443, the new record is: BlockHeader(block_number=17285423, block_hash='0x8d481922bd607150c9f3299004a113e44955327770ab04ed10de115e2172d6fe', timestamp=1684400615)
Python stacktrace:
 [1] add_block
   @ eth_defi.event_reader.reorganisation_monitor ~/Library/Caches/pypoetry/virtualenvs/cryptopy-5siZoxZ4-py3.10/lib/python3.10/site-packages/eth_defi/event_reader/reorganisation_monitor.py:324
 [2] figure_reorganisation_and_new_blocks
   @ eth_defi.event_reader.reorganisation_monitor ~/Library/Caches/pypoetry/virtualenvs/cryptopy-5siZoxZ4-py3.10/lib/python3.10/site-packages/eth_defi/event_reader/reorganisation_monitor.py:396
 [3] update_chain

[hieuh25]

How do you only load the last N blocks? This code wants to loads only last 5 blocks i believe, however it errors when adding new blocks.

    reorg_mon = JSONRPCReorganisationMonitor(web3, check_depth=30)

    reorg_mon.load_initial_block_headers(block_count=5)

    while True:
        try:
            # Figure out the next good unscanned block range,
            # and fetch block headers and timestamps for this block range
            chain_reorg_resolution = reorg_mon.update_chain()

            if chain_reorg_resolution.reorg_detected:
                logger.info(f"Chain reorganisation data updated: {chain_reorg_resolution}")

            # Read specified events in block range
            for log_result in read_events(
                web3,
                start_block=chain_reorg_resolution.latest_block_with_good_data + 1,
                end_block=chain_reorg_resolution.last_live_block,
                filter=filter,
                notify=None,
                chunk_size=100,
                context=token_cache,
                extract_timestamps=None,
                reorg_mon=reorg_mon,
            ):
               pass

INFO:eth_defi.event_reader.reorganisation_monitor:figure_reorganisation_and_new_blocks(), range 17,285,423 - 17,285,443, last block we have is 17,285,443, check depth is 20
ERROR: LoadError: Python: AssertionError: Blocks must be added in order. Last block we have: 17285443, the new record is: BlockHeader(block_number=17285423, block_hash='0x8d481922bd607150c9f3299004a113e44955327770ab04ed10de115e2172d6fe', timestamp=1684400615)
Python stacktrace:
 [1] add_block
   @ eth_defi.event_reader.reorganisation_monitor ~/Library/Caches/pypoetry/virtualenvs/cryptopy-5siZoxZ4-py3.10/lib/python3.10/site-packages/eth_defi/event_reader/reorganisation_monitor.py:324
 [2] figure_reorganisation_and_new_blocks
   @ eth_defi.event_reader.reorganisation_monitor ~/Library/Caches/pypoetry/virtualenvs/cryptopy-5siZoxZ4-py3.10/lib/python3.10/site-packages/eth_defi/event_reader/reorganisation_monitor.py:396
 [3] update_chain

Thank you for reporting this to us!

The reason your script failed was the check_depth is 30, which is higher than initial loaded block count 5.
So what happened was:

• The reorganisation monitor loaded 5 block headers into it's block map
• The while loop started, it ran reorg_mon.update_chain(), which tried to check the specified depth of 30 last read blocks:

  web3-ethereum-defi/eth_defi/event_reader/reorganisation_monitor.py

  Lines 383 to 386 in d47802c

  	chain_last_block = self.get_last_block_live()
  	check_start_at = max(self.last_block_read - self.check_depth, 1)
  	logger.info(f"figure_reorganisation_and_new_blocks(), range {check_start_at:,} - {chain_last_block:,}, last block we have is {self.last_block_read:,}, check depth is %d", self.check_depth)

• The block range it tried to read didn't exist in block map, so it tried to add and failed

This is more or less undocumented feature, so I will either add a warning for this particular case or add some assert to prevent it to happen

[miohtama]

I just close this as there has not been any further comments.