Indicators Optimization
xandortelvanyx opened this issue · 1 comments
xandortelvanyx commented
In the past 15 days, I have watched Mr. Chad Thackray's playlist on YouTube twice and read the Backtesting.py API Reference Documentation. I primarily work with dictionaries because I find NumPy and pandas a bit complicated, so my experience is working on dictionaries.
So far, I have solved many problems and successfully integrated my strategy into Backtesting.py. However, I am currently facing a final issue that I have tried to resolve in every possible way, but I still cannot solve it.
Functions
prepare()
Initializes the data for Backtesting.py.
ohlc()
Retrieves chart data.
indicators()
Obtains analysis data.
TestAnalytics.start()
Initializes the main analytics core.
TestAnalytics.run()
- Retrieves live data or data from a CSV file.
- Applies strategies and indicators.
- Saves the processed data in
data.csvand returns it.
data.csv Preview
| time | low | high | open | close | volume | spread | type | pattern | high_tail | low_tail | tails | body | diff | all | fair_value_gap | support_and_resistance | market_structure_shift | break_of_structure | rsi | sma | signal_buy | signal_sell | signal_neutral | action | stop_loss | limit_price | take_profit | tp_pip | sl_pip |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1698105600 | 1.06609 | 1.06734 | 1.06686 | 1.06721 | 4693 | 1 | 1.06436 | ||||||||||||||||||||||
| 1698134400 | 1.06439 | 1.06947 | 1.06789 | 1.06501 | 17555 | 0 | resistance | HH | up | 1.0667 | 2 | 0 | 1 | buy | 1.04624 | 1.05563 | 1.06501 | ||||||||||||
| 1698148800 | 1.06122 | 1.0651 | 1.06497 | 1.06158 | 14720 | 0 | 1.06572 | ||||||||||||||||||||||
| 1698163200 | 1.05829 | 1.06243 | 1.06157 | 1.05866 | 24104 | 0 | 1.06407 | ||||||||||||||||||||||
| 1698177600 | 1.05844 | 1.05941 | 1.05867 | 1.05886 | 7136 | 0 | 1.0624 |
optimaze.py Code
from backtesting import Backtest, Strategy
from app.analyst.testanalytics import TestAnalytics
import pandas as pd
import numpy as np
TestAnalytics.start(
login=5030094614,
password='DlCg+gH0',
server='MetaQuotes-Demo'
)
def prepare(data):
df = pd.DataFrame(data[1:], columns=data[0]) if isinstance(data, list) else data
df.rename(columns={
'time': 'Date',
'low': 'Low',
'high': 'High',
'open': 'Open',
'close': 'Close',
'volume': 'Volume'
}, inplace=True)
df.replace('', np.nan, inplace=True)
return df
def ohlc(support_and_resistance_candleback=18, sma_candleback=20, rsi_candleback=14):
raw_data = TestAnalytics.run(
support_and_resistance_candleback=support_and_resistance_candleback,
sma_candleback=sma_candleback,
rsi_candleback=rsi_candleback
)
df = prepare(raw_data)
required_columns = ['Date', 'Low', 'High', 'Open', 'Close', 'Volume']
df = df[required_columns]
df.replace('', np.nan, inplace=True)
df['Date'] = pd.to_numeric(df['Date'], errors='coerce')
df['Date'] = pd.to_datetime(df['Date'], unit='s', errors='coerce')
df.set_index('Date', inplace=True)
numeric_columns = ['Low', 'High', 'Open', 'Close', 'Volume']
df[numeric_columns] = df[numeric_columns].apply(pd.to_numeric, errors='coerce')
df.to_csv('_debug/ohlc.csv', index=False)
# print(df)
return df
def indicators(support_and_resistance_candleback=18, sma_candleback=20, rsi_candleback=14):
raw_data = TestAnalytics.run(
support_and_resistance_candleback=support_and_resistance_candleback,
sma_candleback=sma_candleback,
rsi_candleback=rsi_candleback
)
df = prepare(raw_data)
required_columns = ['action', 'limit_price', 'stop_loss', 'take_profit']
df = df[required_columns]
df.replace('', pd.NA, inplace=True)
numeric_columns = ['limit_price', 'stop_loss', 'take_profit']
df[numeric_columns] = df[numeric_columns].apply(pd.to_numeric, errors='coerce')
if 'action' in df.columns:
action_mapping = {'buy': 1, 'sell': -1, 'neutral': 0}
df['action'] = df['action'].replace(action_mapping)
for col in required_columns:
df[col] = df[col].fillna(0)
df['action'] = df['action'].astype(int)
df.to_csv('_debug/indicators.csv', index=False)
# print(df)
return df
class LimitOrderStrategy(Strategy):
trade_size = 1000
support_and_resistance_candleback = 18
sma_candleback = 20
rsi_candleback = 14
counter = 0
def init(self):
self.counter = 0
self.indicators = self.I(
indicators,
self.support_and_resistance_candleback,
self.sma_candleback, self.rsi_candleback
)
np.set_printoptions(threshold=np.inf)
indicator_array = np.array(self.indicators.data)
# print(indicator_array)
if indicator_array.shape[0] == 4:
indicator_array = indicator_array.T
if indicator_array.size > 0 and not np.all(np.isnan(indicator_array)):
indicator_data = pd.DataFrame(
indicator_array,
columns=['action', 'limit_price', 'stop_loss', 'take_profit']
)
indicator_data.to_csv('_debug/indicators.csv', index=False)
# print(indicator_data)
self.actions = indicator_data['action'].to_numpy()
self.limit_prices = indicator_data['limit_price'].to_numpy()
self.stop_losss = indicator_data['stop_loss'].to_numpy()
self.take_profits = indicator_data['take_profit'].to_numpy()
# print(self.actions)
def next(self):
action = int(self.actions[self.counter])
limit_price = self.limit_prices[self.counter]
stop_loss = self.stop_losss[self.counter]
take_profit = self.take_profits[self.counter]
# print(action)
if action in [1, -1]:
if action == 1:
self.buy(size=self.trade_size, limit=limit_price, stop=stop_loss, tp=take_profit)
elif action == -1:
self.sell(size=self.trade_size, limit=limit_price, stop=stop_loss, tp=take_profit)
self.counter += 1
def main():
df = ohlc()
bt = Backtest(
df,
LimitOrderStrategy,
cash=10000,
commission=0.000,
trade_on_close=False,
exclusive_orders=True
)
stats, heatmap = bt.optimize(
sma_candleback=range(5, 20),
maximize='Win Rate [%]',
return_heatmap=True
)
print(stats)
print(heatmap)
if __name__ == "__main__":
main()
Debugging
ohlc() return
Low High Open Close Volume
Date
2023-09-26 00:00:00 1.05805 1.05963 1.05928 1.05842 2786
2023-09-26 04:00:00 1.05781 1.05929 1.05841 1.05926 3077
2023-09-26 08:00:00 1.05700 1.06000 1.05926 1.05959 8667
2023-09-26 12:00:00 1.05895 1.06091 1.05960 1.06058 7775
2023-09-26 16:00:00 1.05674 1.06065 1.06058 1.05692 12413
... ... ... ... ... ...
2024-01-22 04:00:00 1.08979 1.09093 1.09018 1.09015 13536
2024-01-22 08:00:00 1.08871 1.09096 1.09015 1.08970 29556
2024-01-22 12:00:00 1.08797 1.08994 1.08970 1.08917 36801
2024-01-22 16:00:00 1.08816 1.09000 1.08917 1.08961 43915
2024-01-22 20:00:00 1.08801 1.08965 1.08961 1.08820 23934
[499 rows x 5 columns]
indicators() return
action limit_price stop_loss take_profit
0 0 0.0 0.0 0.0
1 0 0.0 0.0 0.0
2 0 0.0 0.0 0.0
3 0 0.0 0.0 0.0
4 0 0.0 0.0 0.0
.. ... ... ... ...
494 0 0.0 0.0 0.0
495 0 0.0 0.0 0.0
496 0 0.0 0.0 0.0
497 0 0.0 0.0 0.0
498 0 0.0 0.0 0.0
[499 rows x 4 columns]
indicator_array in init()
[[ 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 1. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. -1. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
1. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 1. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 1. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. -1. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 1.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. -1.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. ]
[ 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 1.05563 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 1.06094 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
1.06579 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 1.07624 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 1.07872 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 1.08764 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 1.09382
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 1.09982
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. ]
[...][...]]
indicator_data in init()
action limit_price stop_loss take_profit
0 0.0 0.0 0.0 0.0
1 0.0 0.0 0.0 0.0
2 0.0 0.0 0.0 0.0
3 0.0 0.0 0.0 0.0
4 0.0 0.0 0.0 0.0
.. ... ... ... ...
494 0.0 0.0 0.0 0.0
495 0.0 0.0 0.0 0.0
496 0.0 0.0 0.0 0.0
497 0.0 0.0 0.0 0.0
498 0.0 0.0 0.0 0.0
[499 rows x 4 columns]
self.actions in init()
[ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. -1.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. -1. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. -1. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
action in next()
Using [-1] in next()
All values are 0
0
0
0
0
0
0
0
0
0
0
...
Using self.counter in next()
The values are correct, but they do not update with each change in sma_candleback during optimization.
0
0
1
0
0
0
0
0
-1
0
Output
Using [-1] in next()
Code
action = int(self.actions[[-1]])
limit_price = self.limit_prices[[-1]]
stop_loss = self.stop_losss[[-1]]
take_profit = self.take_profits[[-1]]
Result
Start 2023-09-26 00:00:00
End 2024-01-22 20:00:00
Duration 118 days 20:00:00
Exposure Time [%] 0.0
Equity Final [$] 10000.0
Equity Peak [$] 10000.0
Return [%] 0.0
Buy & Hold Return [%] 2.813628
Return (Ann.) [%] 0.0
Volatility (Ann.) [%] 0.0
Sharpe Ratio NaN
Sortino Ratio NaN
Calmar Ratio NaN
Max. Drawdown [%] -0.0
Avg. Drawdown [%] NaN
Max. Drawdown Duration NaN
Avg. Drawdown Duration NaN
# Trades 0
Win Rate [%] NaN
Best Trade [%] NaN
Worst Trade [%] NaN
Avg. Trade [%] NaN
Max. Trade Duration NaN
Avg. Trade Duration NaN
Profit Factor NaN
Expectancy [%] NaN
SQN NaN
_strategy LimitOrderStrate...
_equity_curve ...
_trades Empty DataFrame
...
dtype: object
sma_candleback
1 NaN
2 NaN
3 NaN
4 NaN
5 NaN
6 NaN
7 NaN
8 NaN
9 NaN
10 NaN
11 NaN
12 NaN
13 NaN
14 NaN
15 NaN
16 NaN
17 NaN
18 NaN
19 NaN
Name: Win Rate [%], dtype: float64
Using self.counter in next()
Code
action = int(self.actions[[self.counter]])
limit_price = self.limit_prices[[self.counter]]
stop_loss = self.stop_losss[[self.counter]]
take_profit = self.take_profits[[self.counter]]
Result
Start 2023-09-26 00:00:00
End 2024-01-22 20:00:00
Duration 118 days 20:00:00
Exposure Time [%] 99.398798
Equity Final [$] 10030.01
Equity Peak [$] 10053.34
Return [%] 0.3001
Buy & Hold Return [%] 2.813628
Return (Ann.) [%] 0.975993
Volatility (Ann.) [%] 0.786113
Sharpe Ratio 1.241542
Sortino Ratio 1.970413
Calmar Ratio 3.542233
Max. Drawdown [%] -0.27553
Avg. Drawdown [%] -0.069088
Max. Drawdown Duration 25 days 12:00:00
Avg. Drawdown Duration 5 days 18:00:00
# Trades 1
Win Rate [%] 100.0
Best Trade [%] 2.832201
Worst Trade [%] 2.832201
Avg. Trade [%] 2.832201
Max. Trade Duration 118 days 08:00:00
Avg. Trade Duration 118 days 08:00:00
Profit Factor NaN
Expectancy [%] 2.832201
SQN NaN
_strategy LimitOrderStrate...
_equity_curve ...
_trades Size EntryBa...
dtype: object
sma_candleback
1 100.0
2 100.0
3 100.0
4 100.0
5 100.0
6 100.0
7 100.0
8 100.0
9 100.0
10 100.0
11 100.0
12 100.0
13 100.0
14 100.0
15 100.0
16 100.0
17 100.0
18 100.0
19 100.0
Name: Win Rate [%], dtype: float64
Other attempts
-
I tried assigning the data to sefla.data in init(), but it doesn't work because the library takes the data only at the first initialization, not during every optimization loop.
-
I'm trying to use
self.counterbut I think it's not working.
Goal
I want to optimize my strategies and indicators.
kernc commented
I think you fail to mention what exactly the issue is, but if you mean to have self.counter match the number of iterations, you can use counter = len(self.data)?
Do note that the first iteration the method next() is called varies, dependent on your indicators' warm-up lengths (see warning on Backtest.run).