Decompression speed improvements
mhluska opened this issue · 0 comments
mhluska commented
Currently when loading a compressed solution file, it takes very long because zstd doesn't support multi-threaded decompression. Only compression.
One way around this would be to instead serialize into multiple files instead of one large file. Particularly the Vec[u8]
data from storage1
and node_arena
which can then be deserialized in parallel into a single vector in memory.
It looks like pzstd uses a similar strategy.
I can submit a PR if the library author is interested. I have something working now which looks promising:
- loading a 6GB solution compressed with zstd level 3: 10s
- loading the same solution split into file chunks compressed with zstd level 3, decompressed in parallel: 3.9s