Current data export mechanism relies too heavily on workers being error free
Closed this issue · 1 comments
GrayedFox commented
The current logic of the program writes a block, and all of its transactions, in a single write and retains the scraped order when writing to the exportedData.csv file.
While this is good for later data analysis (and saves us having to sort hundreds of millions of lines later on) the write mechanism will wait until a worker sends it the next descending block before it writes, which can result in the following bug:
- master process writes block 1000 and all transactions to file
- worker 1 finishes block 1002, sending required data to master
- worker 2, 3, and 4 are still busy with blocks 1001, 1003, 1004
- master stores block 1002 data in memory, tells worker 1 to go to next available block
- worker 1 starts scraping block 1005
- worker 3 and 4 finish, master also adds their blocks to memory (block 1003 and 1004), since it hasn't received block 1001 yet, and start scraping next blocks
- worker 2 dies/freezes/poos for some reason, so block 1001 is never sent to master
- remaining workers continue to scrape next available blocks but master never restarts worker 1 and (currently) will never actually write any more data, since worker 1 is never restarted and remaining workers will continue to scrape blocks in order of blockHeight
This can result in the scraper doing all of it's good work but never exporting new data since it refuses to write blocks out of order.
Possible fixes:
- allow blocks to be written and skip a block sometimes if the blocksToWrite array is getting too full (more of a workaround)
- make workers smarter so that they will only scrape the next available block if a previous block is not available to work on (this case should only happen when a worker errors or dies)
- above solution would require proper error handling so that workers save failed blocks somewhere