CIRADA-Tools/RM-Tools

do_fitIcube: tqdm.contrib.concurrent doesn't play nicely with Prefect

ErikOsinga opened this issue · 0 comments

It turns out that the combination of Docker + Prefect + Multiprocessing results in a deadlock after all spectra that are passed to do_fitIcube.make_model_I have been fit.

This is problematic for processing tiles on the CANFAR system, because the first two of those three components are the way we have set it up so far. It seems that the way processes are launched inside a docker container (using fork instead of spawn is causing the issue: PrefectHQ/prefect#10794 (comment))

Though I'm not fully sure what that means, I did figure out how to fix it:

This line is causing issues, which is using from tqdm.contrib.concurrent import process_map to create a progress bar for parallel processing.

"""
results = process_map(
    func,
    srcData,
    max_workers=num_cores,
    chunksize=chunk_size,
    disable=not verbose,
    desc="Fitting spectra",
    total=nDetectPix,
)
"""

However, the progress bar is only used if verbose=True. I propose that if verbose=False we do launch the multiprocessing in the following way

import multiprocessing as mp
mp.set_start_method('spawn')
args_list = [d for d in srcData]
with mp.Pool(processes=num_cores) as pool:
    results = pool.map(func, args_list)

I'll implement this in a PR