Question: What is the best way to concatenate compressed data chunks?
bill-poole opened this issue · 2 comments
I have a system whereby several small chunks of data are compressed and stored, and then those chunks must be then aggregated into a larger single Zstandard compressed image to be loaded into a third party application. Is there a more efficient way of doing this than decompressing all the chunks and then re-compressing the result?
The documentation for the zstd tool states, "It is possible to concatenate multiple .zst files. zstd will decompress such agglomerated file as if it was a single .zst file." However, I am using the ZstdSharp port of this library, so would like to know if it is possible using this library (as opposed to the command line tool) to create a single zstd compressed image from a set of compressed images without decompressing and recompressing.
While I'm not familiar with ZstdSharp specifically,
it's likely that this binding layer follows good practices,
and therefore outputs a payload which is directly compatible with any zstd decoder.
In that case, just concatenating multiple compressed frames produced by ZstdSharp
will result in an agglomerate that zstd is able to decode in one go as if it was a single big frame.
It shouldn't be too difficult to test that property with a toy example and find out.
It turns out that I had a bug in the test that I did before I raised this issue. Apologies for the inconvenience.