Replace parallel system
Closed this issue · 1 comments
The standard multiprocessing library forces a lot of awkward workarounds in wxee e.g. aliased instance methods, using functools to pass args, zipping args, etc. It slows down development and won't scale well.
There are a variety of non-standard library packages for parallelizing that may do a better job like Ray and Dask. I need to decide if there's a better option available and implement it if so.
To avoid adding dependencies (or if a conda-forge recipe is not available), I may use a soft dependency where downloads will run in serial if the package cannot be imported, but the user will be notified and instructed on how to download.
See the pysal guide for a reference on managing soft dependencies that would be especially relevant to Ray, which uses a lot of decorators.