b-inary/postflop-solver

Decompression speed improvements

mhluska opened this issue · 0 comments

Currently when loading a compressed solution file, it takes very long because zstd doesn't support multi-threaded decompression. Only compression.

One way around this would be to instead serialize into multiple files instead of one large file. Particularly the Vec[u8] data from storage1 and node_arena which can then be deserialized in parallel into a single vector in memory.

It looks like pzstd uses a similar strategy.

I can submit a PR if the library author is interested. I have something working now which looks promising:

  • loading a 6GB solution compressed with zstd level 3: 10s
  • loading the same solution split into file chunks compressed with zstd level 3, decompressed in parallel: 3.9s