do_fitIcube: tqdm.contrib.concurrent doesn't play nicely with Prefect
ErikOsinga opened this issue · 0 comments
It turns out that the combination of Docker + Prefect + Multiprocessing results in a deadlock after all spectra that are passed to do_fitIcube.make_model_I
have been fit.
This is problematic for processing tiles on the CANFAR system, because the first two of those three components are the way we have set it up so far. It seems that the way processes are launched inside a docker container (using fork
instead of spawn
is causing the issue: PrefectHQ/prefect#10794 (comment))
Though I'm not fully sure what that means, I did figure out how to fix it:
This line is causing issues, which is using from tqdm.contrib.concurrent import process_map
to create a progress bar for parallel processing.
"""
results = process_map(
func,
srcData,
max_workers=num_cores,
chunksize=chunk_size,
disable=not verbose,
desc="Fitting spectra",
total=nDetectPix,
)
"""
However, the progress bar is only used if verbose=True
. I propose that if verbose=False
we do launch the multiprocessing in the following way
import multiprocessing as mp
mp.set_start_method('spawn')
args_list = [d for d in srcData]
with mp.Pool(processes=num_cores) as pool:
results = pool.map(func, args_list)
I'll implement this in a PR