Multiprocessing Error When Creating Portofiolio
daviddwlee84 opened this issue · 1 comments
daviddwlee84 commented
When I tried to parallelize the backtesting, I found two different kinds of error
First, seems VectorBT Portfolio is not (built-in) pickable AttributeError: Can't pickle local object 'cached_method.<locals>.decorator.<locals>.wrapper.<locals>.partial_func'
(but somehow it can be saved as pickle pf.save()
)
Seconds, if use a package like pathos
to bypass the pickling issue, seems the extended VectorBT stuff may not initialize normally AttributeError: 'Config' object has no attribute '_readonly_'
vectorbt/vectorbt/utils/config.py
Lines 482 to 485 in 67da37e
Here is how to reproduce the error
# 1. Import packages, create data, and try basic operations
import pandas as pd
import numpy as np
import vectorbt as vbt
from tqdm import tqdm
tqdm.pandas()
from pandarallel import pandarallel
pandarallel.initialize(progress_bar=True)
seires_of_dfs = pd.Series(
[
pd.DataFrame(
{"close": np.abs(np.random.randn(10)), "signal": np.random.randn(10)}
)
for _ in range(10)
]
)
def add(df: pd.DataFrame) -> pd.Series:
return df["close"] + df["signal"]
print(seires_of_dfs.apply(add))
print(seires_of_dfs.progress_apply(add))
print(seires_of_dfs.parallel_apply(add))
# Multiprocessing
def backtest(df: pd.DataFrame) -> vbt.Portfolio:
long_signal = df["signal"] > 0.5
short_signal = df["signal"] < -0.5
return vbt.Portfolio.from_signals(
close=df["close"], entries=long_signal, exits=short_signal
)
# fine
print(seires_of_dfs.progress_apply(backtest))
# BUG: AttributeError: Can't pickle local object 'cached_method.<locals>.decorator.<locals>.wrapper.<locals>.partial_func'
print(seires_of_dfs.parallel_apply(backtest))
# Try multi-process pool
from multiprocessing import Pool
# or
from pathos.pools import _ProcessPool as Pool
with Pool(60) as pool:
futures = []
results = []
for df in seires_of_dfs:
futures.append(pool.apply_async(backtest, args=(df,)))
for future in futures:
# BUG (when using multiprocessing): multiprocessing.pool.MaybeEncodingError: Error sending result: '<vectorbt.portfolio.base.Portfolio object at 0x7f613c772040>'. Reason: 'AttributeError("Can't pickle local object 'cached_method.<locals>.decorator.<locals>.wrapper.<locals>.partial_func'")'
# BUG (when using pathos): AttributeError: 'Config' object has no attribute '_readonly_'
results.append(future.get())
Version
- Python 3.8.13
- vectorbt 0.26.1
- pathos 0.3.2
- pandas 1.5.3
- numpy 1.23.5
- pandarallel 1.6.5