very slow i/o on arc
drtobybrown opened this issue · 0 comments
drtobybrown commented
Read/writing data on arc is ~30x slower than on MacBook Pro
MacBook Pro (14-inch, 2021)
» wget https://gist.githubusercontent.com/Jimilian/29a11c9ba799ebd77abf129e7339422e/raw/bb9a95e863a5d8e86fc36a1c829741b880d63ce4/perf_test_serializators.py
» python perf_test_serializators.py
iterations= 100000
data= 'avzvasdklfjhaskldjfhkweljrqlkjb*@&$Y)(!#&$G@#lkjabfsdflb(*!G@#$(GKLJBmnz,bv(PGDFLKJ'
==== DUMP ====
Pickle:
>> 0.02323204200000001
Json: 2.0.9
>> 0.052325416999999985
Marshal: 4
>> 0.015486083999999983
Msgpack: (1, 0, 4)
>> 0.020649415999999976
==== LOAD ====
Pickle:
>> 0.014170584000000042
Json: 2.0.9
>> 0.08166637500000001
Marshal: 4
>> 0.00776574999999996
Msgpack: (1, 0, 4)
>> 0.009213915999999989
science platform arc/
...
==== DUMP ====
Pickle:
>> 0.20738047594204545
Json: 2.0.9
>> 0.8866461659781635
Marshal: 4
>> 0.30416943319141865
Msgpack: (1, 0, 3)
>> 0.4073137487284839
==== LOAD ====
Pickle:
>> 0.39129116013646126
Json: 2.0.9
>> 1.80569237889722
Marshal: 4
>> 0.2898394321091473
Msgpack: (1, 0, 3)
>> 0.2016866127960384