error message thrown despite files copying

[yvan]

(smapputil) [yns207@c26-05 smapputil]$ python py/olympus_2_scratch/olympus2scratch.py -c us_circadian_march_madness -n 4
files will be stored here: /scratch/yns207/olympus_local/us_circadian_march_madness
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/yns207/anaconda/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/home/yns207/anaconda/lib/python3.6/multiprocessing/pool.py", line 47, in starmapstar
return list(itertools.starmap(args[0], args[1]))
File "py/olympus_2_scratch/olympus2scratch.py", line 86, in copy_unzip_clean
clean_file(f_u)
File "py/olympus_2_scratch/olympus2scratch.py", line 57, in clean_file
clean_tweets(f, clean, dirty)
File "/home/yns207/anaconda/lib/python3.6/site-packages/smappdragon/tools/tweet_cleaner.py", line 8, in clean_tweets
json_handle = open(input_file_path, 'r', encoding='utf-8')
FileNotFoundError: [Errno 2] No such file or directory: '/scratch/yns207/olympus_local/us_circadian_march_madness/us_circadian_march_madness_data__04_12_2017__00_00_00__23_59_59.json'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "py/olympus_2_scratch/olympus2scratch.py", line 114, in
main()
File "py/olympus_2_scratch/olympus2scratch.py", line 111, in main
pool.starmap(copy_unzip_clean, args)
File "/home/yns207/anaconda/lib/python3.6/multiprocessing/pool.py", line 268, in starmap
return self._map_async(func, iterable, starmapstar, chunksize).get()
File "/home/yns207/anaconda/lib/python3.6/multiprocessing/pool.py", line 608, in get
raise self._value
FileNotFoundError: [Errno 2] No such file or directory: '/scratch/yns207/olympus_local/us_circadian_march_madness/us_circadian_march_madness_data__04_12_2017__00_00_00__23_59_59.json'

[yvan]

leon says:

change the pool to this:
for f in files:
copy_unzip(f, collection_local)

[yvan]

seems to copy all bzip2 files and unzip them and (hopefully) clean them. so im not overly concerned about the error.

[yinleon]

The issue was that the bunzip step -- function here, does not have STDOUT that waits to resolve before moving on.

We can decompress another way, or find another way to communicate.

In the meantime fixed, by waiting for the compressed file to disappear (which occurs when decompression is complete.)