Parallelize writing of CC data
carlosgjs opened this issue · 0 comments
carlosgjs commented
Testing ZarrCCStore writing on our machine. The writing costs 9.39s for ASDF (local drive) while 89.46s for Zarr (to a S3-like local cloud storage). Monitoring the writing, seems the writing is single threaded that only one channel pair is written at a time. Should we also parallelize this part? https://github.com/mdenolle/NoisePy/blob/47d2ec48331a4d3f4ed787c71877a5cefa6b95a8/src/noisepy/seis/S1_fft_cc_MPI.py#L168-L185