/mecab-parallel-experiment

Parallel processing for a single large text file with MeCab

Primary LanguagePython

M1 MacBook pro (10 cores) mean std min median max improvement
naive 162.568 0.895 161.700 162.386 164.721 x1.000
serial_read 42.620 0.620 42.148 42.428 44.357 x3.814
parallel_read 35.527 0.106 35.345 35.511 35.732 x4.576
parallel_read_write 20.826 0.255 20.598 20.732 21.523 x7.806
parallel_read_write_at_once 20.678 0.146 20.493 20.645 21.036 x7.862
Ubuntu Server (40 cores, NFS) mean std min median max improvement
naive 444.190 47.739 404.738 429.977 579.257 x1.000
serial_read 350.161 207.625 139.832 276.194 749.342 x1.269
parallel_read 278.844 229.585 61.472 179.473 719.069 x1.593
parallel_read_write 52.463 2.525 49.139 51.760 56.053 x8.467
parallel_read_write_at_once 56.320 11.061 46.864 52.601 82.210 x7.887