SimonBlanke/Hyperactive

Progress Bar visual error when running in parallel

SimonBlanke opened this issue · 2 comments

SimonBlanke commented

The tqdm-based progress-bar shows visual errors when running in parallel (with multiprocessing pool). It appears as if new progress-bars are created but not finished. I am not sure if this needs to be fixed in Hyperactive or tqdm.

This problem seems to be related to the following issues tqdm/tqdm#811, tqdm/tqdm#285

I tried a few things to solve this but nothing worked:

Passing a lock to multiprocessing Pool:

Pool(initializer=tqdm.set_lock, initargs=(tqdm.get_lock(),))

If I initialize the progress-bars before starting the multiprocessing by passing them to the parallel function I get the following error:

TypeError: cannot pickle '_io.TextIOWrapper' object
SimonBlanke commented

The solution to this problem is two-fold:

  • passing a lock to multiprocessing Pool is the first part
  • the second part seems to be because of this line in the tqdm source-code.

After those two corrections the progress-bars behave like expected.

SimonBlanke commented

I fixed this issue locally by passing a lock, but I cannot do much to fix the tqdm line. I posted a comment in the relevant tqdm-issue.

I will close this issue, because it is not a Hyperactive bug any more. I will post an update if anything on the tqdm side changes.