support fetching of last possible full candle

[planet-winter]

As mentioned in#4, I used ohlcv_batch[-1][0] because some exchanges return an incomplete last candle.

Like pre calculated for the already elapsed time but not fully finished candle. Needs some extra logic to get the correct last candle.

Re-read ccxt docs and fix

[balibou]

Yes but my P/R for issue 2 is not fixing it ? I save the fully finished candle, not the pre calculated one @planet-winter

[balibou]

@planet-winter actually you are right 😅
with my P/R I figured out a small difference for BTCUSDT between my data and binance on tradingview (GMT: Sunday 27 October 2019 12:02:00):

my data:
1572177720000,9305.94,9320.0,9302.93,9302.93,75.352626

binance on tradingview:

[balibou]

Mmmm I'm a bit stuck on that one ...

[planet-winter]

Hi @balibou I did not yet implement distinguishing which exchange does it. Maybe could look at timestamp... With historical data for back testing purposes I would want the complete candle only.

[balibou]

@planet-winter I'm using this repo (https://github.com/Dave-Vallance/bt-ccxt-store) for backtrader (a backtesting tool). One of the features looks interesting:

• Added drop_newest option to avoid loading incomplete candles where exchanges do not support sending ohlcv params to prevent returning partial data

[balibou]

in https://github.com/Dave-Vallance/bt-ccxt-store/blob/c51f95c667ed2f0bafbf38c0ae47df1856bf7b41/samples/kraken-example.py you can find an example with Kraken:

# Get our data
# Drop newest will prevent us from loading partial data from incomplete candles
hist_start_date = datetime.utcnow() - timedelta(minutes=50)
data = store.getdata(dataname='LTC/USD', name="LTCUSD",
                         timeframe=bt.TimeFrame.Minutes, fromdate=hist_start_date,
                         compression=1, ohlcv_limit=50, drop_newest=True) #, historical=True)

[planet-winter]

:) I am using backtrader as well. Thats why I need the data. and I am using the store implementation as well. I probably figured the exchange last candle thing while sifting through this code?

[balibou]

ahah nice ! maybe we can continue talking in a smoother way on telegram ! my id is @balibou

[planet-winter]

Hi @balibou
I am currently trying to handle that with

        if len(ohlcv_batch):
            last_candle = ohlcv_batch[-1]
            last_timestamp = last_candle[0]
            # drop last candle if its timestamp is not older than timeframe
            # some exchanges seemingly provide incomplete candles
            # exchange_milliseconds is a few milliseconds ago by now and 
            if timeframes_minutes_difference(timeframe, exchange_milliseconds, \
                 last_timestamp) < 1:
                 # drop last candle
                 del ohlcv_batch[-1] 

However to correctly handle this for all time frames, considering leap years and eg. February having 28 days, this could be done:

timestamp_now - last_candle_timestamp >= datetime_difference(TimeStamp_now , last_TimeStamp (in timeframe units))

lookup ccxt timeframe conversion to datetime units in a dict

eg.
(TimeStamp_now - TimeStamp_28d_ago) >= TimeStamp(datetime_difference(march_1st_00:00_UTC , 1month(28 days)))

Just an idea dump will look into sometime.

Then sleeping until next fetching should be adjusted as to make fetches for eg. months more timely

[planet-winter]

Hi @balibou
The current push to master solves this issue for me.
I am parsing the ccxt timeframe and checking it is the last current but incomplete candle and then drop it.