tqdm/tqdm

Multiprocessing - Progress bar not updating for starmap

jetlime opened this issue · 0 comments

  • I have marked all applicable categories:
    • exception-raising bug
    • visual output bug
  • I have visited the source website, and in particular
    read the known issues
  • I have searched through the issue tracker for duplicates
  • I have mentioned version numbers, operating system and
    environment, where applicable:

4.66.2 3.9.18 (main, Jan 4 2024, 00:00:00)
[GCC 11.4.1 20230605 (Red Hat 11.4.1-2)] linux

I am performing multi-processing on a python dictionarry using the multiprocessing.starmap() method.

pool = multiprocessing.Pool()

with pool as p:
    with tqdm.tqdm(total=len(dict)) as pbar:
        for _ in p.starmap(func, dict.items()):
            pbar.update()    
            
pool.close()
pool.join()

However, this implementation does not update the progress bar. The bar is either empty during the complete execution of the program:

  0%|                                                                                            | 0/2414 [00:07<?, ?it/s]

Or it is complete, at the end of the execution:

100%|█████████████████████████████████████████████████████████████████████████████████| 2414/2414 [01:42<00:00, 23.53it/s]

I understand it is due to the fact that the starmap() method does not return an iterator. This code would work if using the imap() method though. While using starmap(), what are the possibilities?