parallelize is a Python package to simplify the process of parallelising your taks in Python. It takes advantage of the multiprocessing module to spawn new processes for your job.
- Python 3.X
To install parallelize, you can either install from the source code or using pip.
To install from source code, first clone the repository. Then, run python setup.py install in the root directory.
To build the documentation, run make html inside the docs/ folder. The documention will be found in the docs/build/html directory. Alternatively, view documentation here.
To parallelise a task in Python, you should wrap the entire code inside a function and have the first argument of your function receive the iterable your function will be operating over.
>>> from parallelize import parallelize
>>> def foo(iterable: list) -> int:
... output = 0
... for i in iterable:
... output = i**4
... return output
>>> numbers = list(range(50000000))
>>> %time foo(numbers)
Wall time: 21.5 s
>>> parallelize.parallel(foo, numbers, 6)
Completed 'parallel' in 6.2743 secs