[feature request] Support for zstd compression
Opened this issue · 6 comments
Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Gzip is relatively slow and doesn't compress that well. Zstd seems to be a more modern implementation with great performance and compression levels.
I've wanted to add that to magic-wormhole but the process of getting it standardised is a bit more cumbersome.
Describe the solution you'd like
A clear and concise description of what you want to happen.
Implementing Zstd compression using https://pkg.go.dev/github.com/klauspost/compress/zstd.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Are there any other algorithms that are faster and compress better? I doubt it but it is always worth it to benchmark and compare :)
Additional context
Add any other context or screenshots about the feature request here.
Hello there, and thank you for your issue!
We are currently using pgzip (parallel gzip compression) by @klauspost, which also is the author of compress/zstd
which you have linked. In his own comparison spreadsheet which you can find on the pgzip
and compress
README, the current library (pgzip
) used is faster by a large margin.
I'm not entirely sure if I am reading the results correctly, so please let me know if I didn't. If you find any alternative zstd libraries which are provably faster I'm happy to implement using them as an option!
PS: How fun with another Swede, and with your impressive application Rymdport, it's basically the same name as Portal in Swedish!
Another factor to consider is decompression speed. In that department zstd is significantly better than any gzip/deflate based.
But yeah, zstd is missing a "multithreaded" compression mode. Not sure when I get the time for that.
@klauspost Thanks. That explains things. Which version of compress
is the spreadsheet based on? I try to follow your project closely and I have noticed that there have been a lot of zstd
assembly improvements over the last couple of months.
@Jacalz It is updated about a week ago. So current.
file | algo | level | insize | outsize | millis | mb/s | Reduction | MB/s | |
---|---|---|---|---|---|---|---|---|---|
10gb.tar | pgzip | 1 | 10065157632 | 5168279238 | 2205 | 4352.94 | 48.65% | 4353.23 | |
10gb.tar | s2 | 1 | 10065157632 | 5915541066 | 1026 | 9352.14 | 41.23% | 9355.64 | |
10gb.tar | s2 | 2 | 10065157632 | 5453844650 | 1856 | 5169.32 | 45.81% | 5171.81 | |
10gb.tar | s2 | 3 | 10065157632 | 5192490222 | 31721 | 302.6 | 48.41% | 302.60 | |
10gb.tar | zskp | 1 | 10065157632 | 4907785561 | 22628 | 424.2 | 51.24% | 424.20 | |
10gb.tar | zskp | 2 | 10065157632 | 4624627076 | 33002 | 290.85 | 54.05% | 290.86 | |
10gb.tar | zskp | 3 | 10065157632 | 4426778811 | 62535 | 153.5 | 56.02% | 153.50 | |
10gb.tar | zskp | 4 | 10065157632 | 4198925932 | 382364 | 25.1 | 58.28% | 25.10 |
Decompression speeds:
Algo | MB/s |
---|---|
gzip | 361.11 |
zstd | 1108.13 |
s2 | 7790.85 |
s2 - 1cpu | 1610.48 |
gzip decompression is a bottleneck.
For local network transfer I would personally go for S2. It is made for speed both when compressing and decompressing. But if the expected network speed is below 3 gigabits/s zstd would make more sense.
Thanks for the explanation and clear benchmark results. S2 sounds like a great option then.
Thank you for your input Klaus!
S2 does look like a good overall option indeed. Furthermore, I have noticed that gzip spends a lot of time doing nothing with incompressible/already compressed files, and I read that S2 is great for that usecase.
The best way forward is probably to add an option/flag of using several of these compression algorithms, S2 included, and possibly changing the default in the next major version of Portal. That would require communicating the algorithm used and supported for the sender, and the supported algorithms by the peer.