georgebv/pyextremes

Multiprocessing in MLE model prevents use on AWS Lambda Functions

tch521 opened this issue · 1 comments

(Firstly I'll just say that this is a fantastic library and very easy to use, great job!)

For whatever reason, AWS lambda functions cannot use multiprocessing.Pool(). This is a problem as it is used by pyextremes when fitting an MLE model. So when I attempt to run my AWS lambda function, it crashes, and the current solution is for me to modify line 199 in model_mle.py to if True, thereby ensuring it does not use the multiprocessing approach.

This is not ideal. It's slightly tedious, but AWS describes this method for using multiprocessing that is compatible with lambda functions. (Instead of using Pool you use Process and Pipe with some additional code to set it up.) Alternatively you could just add a kwarg to avoid the multiprocessing method which would be fine too for those of us not concerned with speed, and much easier to implement.

Adding an option like use_multiprocessing which is True by default sounds like a good solution. If you open a PR I'll gladly review and merge.