A simple implementation for embarrassingly parallel for. It allows the backend to be easily switched between sequential (for debugging), multiprocessing (with joblib) or ray. When using Ray, one can share memory between tasks of the for loop, e.g. some context is put once in shared memory instead of being copied over with multiprocessing/joblib.
Here is an example on how to use parfor
in python, you can also check the example in example/demo.py
:
from parfor import parfor
def f(v0: int, v1: float, v2: str) -> str:
return f"{v0}-{v1}-{v2}"
results = parfor(
f,
# list of inputs to call f
[{"v0": 1}, {"v0": 2}, {"v0": 3}],
# context is shared across all calls of f and added the inputs, when using Ray, shared memory is used which is
# faster if the context is large
context=dict(v1="large-array1", v2="large-array2"),
engine='ray', # can be 'sequential', 'joblib', 'futures', 'ray'
)
# ['1-large-array1-large-array2', '2-large-array1-large-array2', '3-large-array1-large-array2']
print(results)
You can also call parfor
by positional arguments instead of passing dictionaries:
results = parfor(
f,
# list of inputs to call f
[[1], [2], [3]],
# context is shared across all calls of f and added the inputs, when using Ray, shared memory is used which is
# faster if the context is large
context=dict(v1="large-array1", v2="large-array2"),
engine='ray',
)
This requires that the positional arguments comes before in the function f
being called.
Here is a potential use-case for each of the backend:
sequential
: great for debuggingfutures
: great for distributing many operations that are IO bound and not compute bound. For instance, querying an external API is a good use-case, parallelizing matrix computation is a bad one.joblib
: great for distributing many operations that are compute bound. For instance, parallelizing matrix computation where little context needs to be shared between tasks.ray
: great for distributing many operations that are compute bound where large context needs to be shared between tasks (for instance sharing a large array, a large object, ...). When using Ray, one can share memory between tasks of the for loop, e.g. some context is put once in shared memory instead of being copied over with multiprocessing/joblib.
pip install git+https://github.com/geoalgo/parfor.git
- fix Ray in CI
- Register in Pypi