Binance periodically fetching initial orderbook snapshot
Closed this issue · 9 comments
When I run BinanceManualExample, in log I see Fetching initial orderbook snapshot for LTC/BTC each four seconds. Is it supposed to be happening so often, or maybe something changed on Binance side so now the logic for fetching initial snapshot does not work correctly? Because a few months ago when I've used it last time, I think this was happening in random intervals and not so often.
It always happened if the market activity was high at the time, and that's all the time now.
I read through Binance's docs when I wrote the code and they don't say re-attempting is the right thing to do, they just say that you know the sync is wrong if the sequence numbers mismatch. Retrying was a quick way to get it working but I think it is struggling with market volume now and a better solution is needed.
I bet there is a way to do this without retrying, by using a recent trades list to reverse those trades.from the snapshot so it precedes the most recent delta. My surprise was that Binance themselves don't suggest anything like this.
Ok I see, it stops happening, after five minutes :)
@badgerwithagun well I looked at the code (with your comments) and at their docs and I think there's a difference between snapshot and stream events in that although events are expected to always be in sequence the snapshot doesn't have to be on the edge. That's why they mention that
The data in each event is the absolute quantity for a price level
Hence, i.e. in my example: Orderbook snapshot for BTC/USDT out of date (last=692812130, U=692812102, u=692812177) - it should not be rerequested because snapshot ID is between first and last update ID hence you can simply apply this update - it will overwrite whatever changes the snapshot already had and that's perfectly fine.
They say it pretty clearly here:
The first processed should have U <= lastUpdateId+1 AND u >= lastUpdateId+1
I.e. the U and u don't have to be equal - just within the range.
The problem with retrying logic is it depends on a chance and the chance can take pretty long to occur...
Hey @dpisklov - that makes sense. I probably misread it!
If that's the case the retry can be safely removed. Will you raise a PR for that? I doubt I'll get around to it quickly.
I hope I'll get a chance to do that - no guarantee though, although this does bother us we are not yet in production so time allows...
The change though is a bit more than removing retry - at the moment in the code there's no distinguishing between stream event and a snapshot, but there should be, i.e. even though we don't need to strictly be in sequence with snapshot + event, we still need to validate sequencing and invalidate local orderbook if the stream events are not in sequence.
(Not sure that was really clear enough but hope it makes some sense :D )
:). Not got the code in front of me, but I get the general idea.
@badgerwithagun PR #334 raised.
On a slightly separate subject, do you plan to add a support for limited-depth (aka partial) books?
No plans, @dpisklov. I only work on this when I have a bit of spare time (I'm only using it for open source projects). Someone else might be thinking about it. Or you can :)
@badgerwithagun got ya, if we need it I'll raise a PR certainly.