Missing order notifications when running live
booboothefool opened this issue · 21 comments
When running my algorithm on a live account with real money, the furthest order notification I seem to get is Submitted -> Accepted.
I have logic that executes after a first take profit target (limit order) is hit, so it cannot activate because I never get the Completed notification for that take profit.
Everything seems to work as expected while backtesting and I get all of the notifications including Completed, but they appear to be missing live.
Then your order is still pending. You will receive completed once the order is executed by oanda.
I am saying what I am seeing is that the order is completed, as in I see it go through on the Oanda dashboard/TradingView, but I never receive the notification in backtrader.
the order is entered (it appears in oanda) but did it execute? like did it open a position?
Yes the orders execute as in there was an open position and I won/lost money, haha.
For example:
self.E1 = self.buy(price=entry_price, size=size, exectype=bt.Order.Stop, transmit=False)
self.SL1 = self.sell(price=stoploss_price, size=size, exectype=bt.Order.Stop, transmit=False, parent=self.E1)
self.TP1 = self.sell(price=TP1_price, size=size, exectype=bt.Order.Limit, transmit=True, parent=self.E1)
E1 is Submitted -> Accepted
E1 is hit and it opens a new position (but no Completed)
TP1 is hit and won money (but no Completed)
could you provide some sample code which shows the problem
Goal is to try to move SL2 to breakeven after TP1 is hit. I am logging out every order notification in the beginning and it never seems to print Completed, so order.status == order.Completed logic never executes.
def notify_order(self, order):
logs = True
logs and print('{}: Order ref: {} / Type {} / Status {} / ExecType {} / Size {} / Alive {} / Price {} / Position {}'.format(
self.data.datetime.date(0),
order.ref,
'Buy' * order.isbuy() or 'Sell',
order.getstatusname(),
order_exectypes[order.exectype],
order.size,
order.alive(),
order.executed.price,
self.position.size,
))
if order.alive():
# Submitted, Accepted, Partial
if order.status in (order.Partial, order.Submitted):
# otherwise these ^ get stuck sometimes
if self.E1 and order.ref == self.E1.ref: self.E1 = None
elif self.E2 and order.ref == self.E2.ref: self.E2 = None
# pass
else:
# Completed, Canceled, Rejected
if self.E1 and order.ref == self.E1.ref: self.E1 = None
elif self.SL1 and order.ref == self.SL1.ref: self.SL1 = None
elif self.TP1 and order.ref == self.TP1.ref:
self.TP1 = None
if self.params.breakeven:
if order.status == order.Completed:
logs and print('HIT TP1')
if self.SL2 and self.last_entry_price:
# if hit TP1, move SL2 to breakeven
logs and print('MOVE SL2 TO BREAKEVEN', self.last_entry_price)
self.move_stop_loss(self.last_entry_price)
self.last_entry_price = None
elif self.E2 and order.ref == self.E2.ref:
self.E2 = None
if self.params.breakeven:
if order.status == order.Completed:
self.last_entry_price = order.executed.price
elif self.SL2 and order.ref == self.SL2.ref:
self.SL2 = None
if self.params.breakeven:
if order.status == order.Completed:
logs and print('CREATED NEW SL2')
# only when the new SL2 gets successfully created, then can we cancel the old one
if self.old_SL2:
logs and print('CANCEL OLD SL2')
self.old_SL2.cancel()
self.old_SL2 = None
elif self.TP2 and order.ref == self.TP2.ref:
self.TP2 = None
if self.params.breakeven:
if order.status == order.Completed:
logs and print('HIT TP2')
# if SL2 was moved to breakeven after TP1, and therefore it is not of the original transmit group, then we now need to manually cancel it after TP2 is hit
if self.SL2 and self.moved_SL2:
logs and print('CANCEL NEW SL2')
self.SL2.cancel()
self.moved_SL2 = False
Goal is to try to move SL2 to breakeven after TP1 is hit. I am logging out every order notification in the beginning and it never seems to get
Completed, soorder.status == order.Completedlogic never executes.def notify_order(self, order): logs = True logs and print('{}: Order ref: {} / Type {} / Status {} / ExecType {} / Size {} / Alive {} / Price {} / Position {}'.format( self.data.datetime.date(0), order.ref, 'Buy' * order.isbuy() or 'Sell', order.getstatusname(), order_exectypes[order.exectype], order.size, order.alive(), order.executed.price, self.position.size, )) if order.alive(): # Submitted, Accepted, Partial if order.status in (order.Partial, order.Submitted): # otherwise these ^ get stuck sometimes if self.E1 and order.ref == self.E1.ref: self.E1 = None elif self.E2 and order.ref == self.E2.ref: self.E2 = None # pass else: # Completed, Canceled, Rejected if self.E1 and order.ref == self.E1.ref: self.E1 = None elif self.SL1 and order.ref == self.SL1.ref: self.SL1 = None elif self.TP1 and order.ref == self.TP1.ref: self.TP1 = None if self.params.breakeven: if order.status == order.Completed: logs and print('HIT TP1') if self.SL2 and self.last_entry_price: # if hit TP1, move SL2 to breakeven logs and print('MOVE SL2 TO BREAKEVEN', self.last_entry_price) self.move_stop_loss(self.last_entry_price) self.last_entry_price = None elif self.E2 and order.ref == self.E2.ref: self.E2 = None if self.params.breakeven: if order.status == order.Completed: self.last_entry_price = order.executed.price elif self.SL2 and order.ref == self.SL2.ref: self.SL2 = None if self.params.breakeven: if order.status == order.Completed: logs and print('CREATED NEW SL2') # only when the new SL2 gets successfully created, then can we cancel the old one if self.old_SL2: logs and print('CANCEL OLD SL2') self.old_SL2.cancel() self.old_SL2 = None elif self.TP2 and order.ref == self.TP2.ref: self.TP2 = None if self.params.breakeven: if order.status == order.Completed: logs and print('HIT TP2') # if SL2 was moved to breakeven after TP1, and therefore it is not of the original transmit group, then we now need to manually cancel it after TP2 is hit if self.SL2 and self.moved_SL2: logs and print('CANCEL NEW SL2') self.SL2.cancel() self.moved_SL2 = False
please post how you create the orders.
def buy_risk(self):
stop_dist = self.set_stop_dist()
entry_dist = self.set_entry_dist()
entry_price = self.last5high[0] + entry_dist
stoploss_price = self.l[0] - stop_dist
R = entry_price - stoploss_price
R_pips_big = R * 10000
TP1_price = entry_price + (1 * R)
TP2_price = entry_price + (2 * R)
size = self.calculate_risk_size(R_pips_big)
print('SL %.4f / Price %.4f / BuyStop %.4f / R=%.4f %.4f / TP1=%.4f / TP2=%.4f / size %.4f / Stop Dist %.4f' % (
stoploss_price,
self.c[0],
entry_price,
R, R_pips_big,
TP1_price,
TP2_price,
size,
stop_dist,
))
self.E1 = self.buy(price=entry_price, size=size, exectype=bt.Order.Stop, transmit=False)
self.SL1 = self.sell(price=stoploss_price, size=size, exectype=bt.Order.Stop, transmit=False, parent=self.E1)
self.TP1 = self.sell(price=TP1_price, size=size, exectype=bt.Order.Limit, transmit=True, parent=self.E1)
self.E2 = self.buy(price=entry_price, size=size, exectype=bt.Order.Stop, transmit=False)
self.SL2 = self.sell(price=stoploss_price, size=size, exectype=bt.Order.Stop, transmit=False, parent=self.E2)
self.TP2 = self.sell(price=TP2_price, size=size, exectype=bt.Order.Limit, transmit=True, parent=self.E2)
ok, will look into this later.
for now you could try if bracket orders will work with your code
I could not really replicate your issue. I entered these values for the orders:
self.E1 = self.buy(price=self.data.close[0], size=100, exectype=bt.Order.Stop, transmit=False)
self.SL1 = self.sell(price=self.data.close[0] + 0.0005, size=100, exectype=bt.Order.Stop, transmit=False, parent=self.E1)
self.TP1 = self.sell(price=self.data.close[0] - 0.0002, size=100, exectype=bt.Order.Limit, transmit=True, parent=self.E1)
but these orders get canceled. you can use the code at the bottom to provide a strategy which replicates your issue.
from __future__ import (absolute_import, division, print_function,
unicode_literals)
# For datetime objects
import datetime
import pytz
# Import the backtrader platform
import backtrader as bt
import btoandav20 as bto
import backtrader.version as btvers
print(btvers.__version__)
class strategy(bt.Strategy):
'''Initialization '''
def __init__(self):
self.live = True
def log(self, txt, dt=None):
dt = dt or self.data.datetime.datetime(0)
print('%s, %s' % (dt.isoformat(), txt))
''' Store notification '''
def notify_store(self, msg, *args, **kwargs):
print('-' * 50, 'STORE BEGIN', datetime.datetime.now())
print(msg)
print('-' * 50, 'STORE END')
''' Order notification '''
def notify_order(self, order):
if order.status in [order.Completed, order.Cancelled, order.Rejected]:
self.order = None
print('-' * 50, 'ORDER BEGIN', datetime.datetime.now())
print(order)
print('-' * 50, 'ORDER END')
''' Trade notification '''
def notify_trade(self, trade):
print('-' * 50, 'TRADE BEGIN', datetime.datetime.now())
print(trade)
print('-' * 50, 'TRADE END')
if not trade.isclosed:
return
self.log('OPERATION PROFIT, GROSS %.2f, NET %.2f' %
(trade.pnl, trade.pnlcomm))
''' Data notification '''
def notify_data(self, data, status, *args, **kwargs):
print('*' * 5, 'DATA NOTIF:', data._getstatusname(status), *args)
if status == data.LIVE:
self.live = True
elif status == data.DELAYED:
self.live = False
def next(self):
self.log('Next {}, {} {} {} {}'.format(len(self.data), self.data.open[0], self.data.high[0], self.data.low[0], self.data.close[0]))
print(self.position)
if not self.position and self.live:
self.E1 = self.buy(price=self.data.close[0], size=100, exectype=bt.Order.Stop, transmit=False)
self.SL1 = self.sell(price=self.data.close[0] + 0.0005, size=100, exectype=bt.Order.Stop, transmit=False, parent=self.E1)
self.TP1 = self.sell(price=self.data.close[0] - 0.0002, size=100, exectype=bt.Order.Limit, transmit=True, parent=self.E1)
elif self.live:
#self.close()
pass
# Create a cerebro entity
cerebro = bt.Cerebro(quicknotify=True)
# Prepare oanda STORE
storekwargs = dict(
token="",
account="",
practice=True,
)
print(bto.stores)
store = bto.stores.OandaV20Store(**storekwargs)
# Prepare oanda data
datakwargs = dict(
timeframe=bt.TimeFrame.Seconds,
compression=5,
tz=pytz.timezone('Europe/Berlin'),
backfill=False,
backfill_start=False,
)
data = store.getdata(dataname="EUR_USD", **datakwargs)
#data.resample(timeframe=bt.TimeFrame.Seconds, compression=30,rightedge=True,boundoff=1)
cerebro.adddata(data)
# Add broker
cerebro.setbroker(store.getbroker())
# Add strategy
cerebro.addstrategy(strategy)
cerebro.run()
Also please Check if the both stop orders (one with entry price) are really stop orders or if the first one should be a limit order.
Hi if you did:
self.E1 = self.buy(price=self.data.close[0], size=100, exectype=bt.Order.Stop, transmit=False)
self.SL1 = self.sell(price=self.data.close[0] + 0.0005, size=100, exectype=bt.Order.Stop, transmit=False, parent=self.E1)
self.TP1 = self.sell(price=self.data.close[0] - 0.0002, size=100, exectype=bt.Order.Limit, transmit=True, parent=self.E1)
It is a Buy position, so SL1 stop loss should be below, TP1 take profit should be above. So maybe just:
self.E1 = self.buy(price=self.data.close[0] + 0.0005, size=100, exectype=bt.Order.Stop, transmit=False)
self.SL1 = self.sell(price=self.data.close[0] - 0.0005, size=100, exectype=bt.Order.Stop, transmit=False, parent=self.E1)
self.TP1 = self.sell(price=self.data.close[0] + 0.0010, size=100, exectype=bt.Order.Limit, transmit=True, parent=self.E1)
And yes, it should be:
- Entry: Buy Stop (to buy when price goes up and hits the price)
- Stop Loss: Sell Stop (below)
- Take Profit: Sell Limit (above)
But from what I have seen, it doesn't matter what the order type is, I never see Completed status when running live even with regular Market order instead of Stop order e.g. self.E1 = self.buy(size=100, exectype=bt.Order.Market, transmit=False).
I am not really sure, how to send this type of orders. When sending your orders, oanda will reject it with CLIENT_ORDER_ID_ALREADY_EXISTS
i committed some changes today, you can check them out. you will get better error codes and descriptions for further investigation. the issue seems to be that either the stores creates the stop order wrong or there is an other issue.
CLIENT_ORDER_ID_ALREADY_EXISTS: The client Order ID specified is already assigned to another pending Order
these are the generated order details:
{'instrument': 'EUR_USD',
'units': 100,
'type': 'STOP',
'price': '1.11979',
'timeInForce': 'GTC',
'stopLossOnFill': {'price': '1.11879', 'timeInForce': 'GTC', 'clientExtensions': {'id': '2'}},
'takeProfitOnFill': {'price': '1.12029', 'timeInForce': 'GTC', 'clientExtensions': {'id': '3'}},
'clientExtensions': {'id': '1'}}
the clientExtensions id is the id of the order in backtrader.
you can check out https://developer.oanda.com/rest-live-v20/transaction-df/ for the needed details for the type of orders you would like to submit.
Would need a better explanation of what you want to achieve. Will wait for more details.
Thanks, I appreciate you exposing the errors! Ok, I will look into this more. Yes, I've been struggling so much with the manual brackets that I just switched to Market orders, and it turns out my algorithm actually performs better...
I heard back from Oanda support. They said what you said:
1) Transaction no.: 40749
Reason for Stop Order Reject: TRAILING_STOP_LOSS_ON_FILL_PRICE_DISTANCE_MINIMUM_NOT_MET
*Please note that Trailing Stop must be at least 5 pips away however you had it at 3 pips away. Hence, it was rejected
2) Transaction no.: 40748, 40747
Reason for Stop Order Reject: CLIENT_ORDER_ID_ALREADY_EXISTS
*Wet cannot create a new Stop order using an order ID that already exists from past orders. Please use a new order ID. (e.g Transaction #: 40747: You are trying to create a Stop order with ID 25 but the order ID 25 already exists from past transaction.).
As for more explanation/details, was basically just trying to do the risk management strategy as shown here: https://youtu.be/zhEukjCzXwM?t=747
we could ensure, that the id is unique, so adding some unique key for id, which changes when you restart the script
As for more explanation/details, was basically just trying to do the risk management strategy as shown here: https://youtu.be/zhEukjCzXwM?t=747
for that, you could achieve this by setting limit orders in opposite direction. you would need to cancel the order once you would set a new order when the price reaches new high (or low).
added a unique client id, now you will not get CLIENT_ORDER_ID_ALREADY_EXISTS messages, for the other issues, Stop orders work, will close this issue.
@ booboothefool
I did not fully go through your post here, but today I will open an issue but you can check my code, I think it is something along the lines of what u want to do.