Look into using zstd for squashfs compression
shvchk opened this issue · 5 comments
shvchk commented
wine-launcher/src/src/modules/pack.js
Line 92 in eb8d096
zstd will probably be a better choice due to much better decompression speed, and it can compress almost as good as xz on higher compression levels
> mksquashfs help
...
-comp <comp> select <comp> compression
...
Compressors available and compressor specific options:
...
zstd
-Xcompression-level <compression-level>
<compression-level> should be 1 .. 22 (default 15)
shvchk commented
Quick comparison of compression speed and ratio with xz on my wine prefix with HoMM 3 (/tmp/x is a ramdisk):
> du -sh wineprefix
723M wineprefix
> for i in 3 6 9 12 15 18 22; do time mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level $i &> /dev/null; du -h /tmp/x/x; rm -f /tmp/x/x; done
mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level $i &> 12,32s user 1,51s system 367% cpu 3,763 total
609M /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level $i &> 55,84s user 1,73s system 374% cpu 15,351 total
605M /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level $i &> 58,90s user 1,77s system 367% cpu 16,506 total
604M /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level $i &> 61,31s user 1,75s system 373% cpu 16,868 total
603M /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level $i &> 77,13s user 1,76s system 307% cpu 25,655 total
603M /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level $i &> 220,34s user 1,56s system 295% cpu 1:15,10 total
598M /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level $i &> 290,87s user 1,82s system 334% cpu 1:27,62 total
598M /tmp/x/x
> time mksquashfs wineprefix /tmp/x/x -b 1M -comp xz &> /dev/null; du -h /tmp/x/x; rm -f /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp xz &> /dev/null 365,15s user 2,01s system 339% cpu 1:48,14 total
589M /tmp/x/x
> time mksquashfs wineprefix /tmp/x/x -b 1M -comp xz -Xdict-size 100% &> /dev/null; du -h /tmp/x/x; rm -f /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp xz -Xdict-size 100% &> /dev/null 366,67s user 1,84s system 342% cpu 1:47,59 total
589M /tmp/x/x
> time mksquashfs wineprefix /tmp/x/x -b 1M -comp xz -Xdict-size 100% -Xbcj x86 &> /dev/null; du -h /tmp/x/x; rm -f /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp xz -Xdict-size 100% -Xbcj x86 &> 731,24s user 2,21s system 324% cpu 3:46,12 total
588M /tmp/x/x
Didn't test decompression speed, as I have no doubt zstd is much better at it.
hitman249 commented
- this command only applies to packaging wine
- the prefix is not compressed at all
- in the choice of format, we are limited to the squashfuse utility
- game data is compressed by this command
wine-launcher/src/src/modules/pack.js
Line 94 in eb8d096
shvchk commented
- Ah, ok. Might be worth it to look at dwarfs if there would be multiple versions of wine and you expect data to be highly redundant.
- Ok, I probably got wrong impression from reading readme
- squashfuse seems to support zstd
- lz4 is nice, but zstd would probably be even better 😄
In tests below, zstd does its job 10+ times faster while compressing better (even on compression level 1):
> du -sh wineprefix
723M wineprefix
> time mksquashfs wineprefix /tmp/x/x -b 1M -comp lz4 -Xhc &> /dev/null; du -h /tmp/x/x; rm -f /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp lz4 -Xhc &> /dev/null 75,28s user 1,24s system 375% cpu 20,389 total
625M /tmp/x/x
> time mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level 1 &> /dev/null; du -h /tmp/x/x; rm -f /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level 1 &> 4,54s user 1,39s system 372% cpu 1,588 total
615M /tmp/x/x
> du -sh ~/.config
707M /home/x/.config
> time mksquashfs ~/.config /tmp/x/x -b 1M -comp lz4 -Xhc &> /dev/null; du -h /tmp/x/x; rm -f /tmp/x/x
mksquashfs ~/.config /tmp/x/x -b 1M -comp lz4 -Xhc &> /dev/null 77,89s user 1,35s system 361% cpu 21,943 total
541M /tmp/x/x
> time mksquashfs ~/.config /tmp/x/x -b 1M -comp zstd -Xcompression-level 1 &> /dev/null; du -h /tmp/x/x; rm -f /tmp/x/x
mksquashfs ~/.config /tmp/x/x -b 1M -comp zstd -Xcompression-level 1 &> 5,37s user 1,40s system 353% cpu 1,914 total
537M /tmp/x/x
hitman249 commented
1. dwarfs has an executable too large compared to squashfuse (16 MB+ vs 535 KB)
4. decompression speed is more critical than compression
shvchk commented
- Might save much more, but I get your point. In this case zstd is still a good alternative to xz.
- Yeah, I guess... lz4hc was ~30% faster at decompression in my in-memory and SSD tests. That compression speed difference though : ) Also, zstd compression ratio is significantly better on higher compression levels. Maybe it's worth to be added as an option some day 😉