pikers/piker

Duplicate history frames can cause inf loop during backfill

goodboy opened this issue · 1 comments

Definitely have seen this on binance where you'll get something like the following for an ltcusdt.binance load:
screenshot-2022-12-20_13-55-22

Likely the fix is to have a max duplicates-per-timestamp in the continue case....

This may have the follow up mega-issue that it corrupts the marketstore column series such that you can't load all the data on the next Storage.read_ohlc() query to this fqsn..

logging will have something like

Dec 20 14:12:37 (brokerd.binance[4733b4], 77469, piker.data.feed.tsdb_backfill.<locals>.back_load_from_tsdb)) [ERROR] piker.data.marketstore marketstore.py:456 Unknown error: ('RPC failed with status UNKNOWN (2): snappy: corrupt input',)

and you'll see only the first (large) frame's worth of data loaded from mkts..