Duplicate history frames can cause inf loop during backfill
goodboy opened this issue · 1 comments
goodboy commented
Definitely have seen this on binance
where you'll get something like the following for an ltcusdt.binance
load:
Likely the fix is to have a max duplicates-per-timestamp in the continue
case....
goodboy commented
This may have the follow up mega-issue that it corrupts the marketstore
column series such that you can't load all the data on the next Storage.read_ohlc()
query to this fqsn..
logging will have something like
Dec 20 14:12:37 (brokerd.binance[4733b4], 77469, piker.data.feed.tsdb_backfill.<locals>.back_load_from_tsdb)) [ERROR] piker.data.marketstore marketstore.py:456 Unknown error: ('RPC failed with status UNKNOWN (2): snappy: corrupt input',)
and you'll see only the first (large) frame's worth of data loaded from mkts..