polakowo/vectorbt

Multiprocessing Error When Creating Portofiolio

daviddwlee84 opened this issue · 1 comments

When I tried to parallelize the backtesting, I found two different kinds of error

First, seems VectorBT Portfolio is not (built-in) pickable AttributeError: Can't pickle local object 'cached_method.<locals>.decorator.<locals>.wrapper.<locals>.partial_func' (but somehow it can be saved as pickle pf.save())

Seconds, if use a package like pathos to bypass the pickling issue, seems the extended VectorBT stuff may not initialize normally AttributeError: 'Config' object has no attribute '_readonly_'

@property
def readonly_(self) -> bool:
"""Whether to deny any updates to the config."""
return self._readonly_

Here is how to reproduce the error

# 1. Import packages, create data, and try basic operations
import pandas as pd
import numpy as np
import vectorbt as vbt

from tqdm import tqdm
tqdm.pandas()

from pandarallel import pandarallel
pandarallel.initialize(progress_bar=True)

seires_of_dfs = pd.Series(
    [
        pd.DataFrame(
            {"close": np.abs(np.random.randn(10)), "signal": np.random.randn(10)}
        )
        for _ in range(10)
    ]
)

def add(df: pd.DataFrame) -> pd.Series:
    return df["close"] + df["signal"]

print(seires_of_dfs.apply(add))
print(seires_of_dfs.progress_apply(add))
print(seires_of_dfs.parallel_apply(add))
# Multiprocessing 
def backtest(df: pd.DataFrame) -> vbt.Portfolio:
    long_signal = df["signal"] > 0.5
    short_signal = df["signal"] < -0.5
    return vbt.Portfolio.from_signals(
        close=df["close"], entries=long_signal, exits=short_signal
    )

# fine
print(seires_of_dfs.progress_apply(backtest))

# BUG: AttributeError: Can't pickle local object 'cached_method.<locals>.decorator.<locals>.wrapper.<locals>.partial_func'
print(seires_of_dfs.parallel_apply(backtest))
# Try multi-process pool
from multiprocessing import Pool
# or
from pathos.pools import _ProcessPool as Pool

with Pool(60) as pool:
    futures = []
    results = []
    for df in seires_of_dfs:
        futures.append(pool.apply_async(backtest, args=(df,)))
    for future in futures:
        # BUG (when using multiprocessing): multiprocessing.pool.MaybeEncodingError: Error sending result: '<vectorbt.portfolio.base.Portfolio object at 0x7f613c772040>'. Reason: 'AttributeError("Can't pickle local object 'cached_method.<locals>.decorator.<locals>.wrapper.<locals>.partial_func'")'
        # BUG (when using pathos): AttributeError: 'Config' object has no attribute '_readonly_'
        results.append(future.get())

Version

  • Python 3.8.13
  • vectorbt 0.26.1
  • pathos 0.3.2
  • pandas 1.5.3
  • numpy 1.23.5
  • pandarallel 1.6.5