Progress bar for multiprocessing

[ulfaslak]

Currently, it's possible to do something like:

def worker(i):
    d = 0
    for _ in range(i):
        d += 1
    return d

data = [int(5e6) for _ in range(10)]

map(worker, log_progress(data, every=1))

If I try to do the same using a multiprocessing mapper, the progress bar is immediately completed.

from multiprocessing import Pool

def worker(i):
    d = 0
    for _ in range(i):
        d += 1
    return d

data = [int(5e6) for _ in range(10)]

p = Pool(2)
processes = p.map(worker, log_progress(data, every=1))
p.close()

Would it be possible to somehow make this work?

[Goddesen]

This is because the Pool.map function gathers the whole iterator into a list before applying the supplied worker function.
Pool.imap might be what you're looking for, as it returns an iterator over the results instead of blocking to create a list with all results at once.

data = log_progress(pool.imap(func, args_list), size=len(args_list))
for e in data:
    ...