Feature request: Output ZPAQ formatted files.
Closed this issue · 4 comments
Most of the time the user would compress a directory instead of a single file, and extract some (not all) of the files in an archive. ZPAQ format allow "solid" archives so you can still have deduplication in the whole archive while you can extract only the files you need. The archive would still be compatible with ZPAQ because the decompression code is already in the archive. Sure it is not streamable but I do not think that being streamable makes too much sense when you archive files.
try tar --use-compress-program="lrzip -z" -cf file.tar.lrz somedir
then tar -I lrzip -xf file.tar.lrz somedir/file
Be sure to quote lrzip -z
But Windows does not have tar. Also not every version of tar has --use-compress-program.
ZPAQ is one single executable.
Lrzip doesn't have any windows support.
The lrzip file format is unique and distinct with its own header. Each stream 0 and stream 1 of each chunk has its own header format as well. No matter what, lrzip files may only be decompressed by lrzip. See Magic Header. If you want a ZPAQ archive, then you must use ZPAQ to create it or run it through tar as explained above. The truth is, the extra time ZPAQ takes to compress and decompress does not compare that favorably over lrzip.
I've done this before, but here is a new analysis comparing lrzip and zpaq (version 7.15. An improved library to that which is used in this version of lrzip. lrzip 0.7x uses the most recent zpaq library).
The following were done using lrzip default options. zpaq using -m4. I've highlighted the best and worst performers.
| File | Size | Compression | Compressed Size | Time to Compress | Compression Ratio | MB/s |
|---|---|---|---|---|---|---|
| linux-5.4.y.tar | 1,639,557,120 | tar | N/A | N/A | N/A | N/A |
| lrzip 0.631 | largest | rzip+lzma | 164,377,635 | 02:19.10 | 9.974 | 11.164 |
| lrzip 0.721 | fastest | rzip+lzma | 159,383,452 | 02:18.21 | 10.287 | 11.326 |
| lrzip 0.631 | rzip+zpaq | 140,330,077 | 04:57.33 | 11.684 | 5.263 | |
| lrzip 0.721 | best | rzip+zpaq | 136,971,418 | 05:49.21 | 11.970 | 4.466 |
| zpaq 7.15 | slowest | zpaq | 153,487,409 | 06:36.40 | 10.682 | 3.693 |
zpaq+rzip provides the best compression, even over zpaq alone. You have to weigh whether the benefits are worth the triple-the-time difference.
Differences in lrzip 0.721 are due to differences in computation of buffer size, lzma sdk, variable lzma dictionary size. YMMV depending on file types.