hitman249/wine-launcher

Look into using zstd for squashfs compression

shvchk opened this issue · 5 comments

cmd = `mksquashfs "${folder}" "${squashfs}" -b 1048576 -comp xz -Xdict-size 100%`;

zstd will probably be a better choice due to much better decompression speed, and it can compress almost as good as xz on higher compression levels

> mksquashfs help
...
-comp <comp>            select <comp> compression
...
Compressors available and compressor specific options:
...
        zstd
          -Xcompression-level <compression-level>
                <compression-level> should be 1 .. 22 (default 15)

Quick comparison of compression speed and ratio with xz on my wine prefix with HoMM 3 (/tmp/x is a ramdisk):

> du -sh wineprefix
723M    wineprefix

> for i in 3 6 9 12 15 18 22; do time mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level $i &> /dev/null; du -h /tmp/x/x; rm -f /tmp/x/x; done
mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level $i &>   12,32s user 1,51s system 367% cpu 3,763 total
609M    /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level $i &>   55,84s user 1,73s system 374% cpu 15,351 total
605M    /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level $i &>   58,90s user 1,77s system 367% cpu 16,506 total
604M    /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level $i &>   61,31s user 1,75s system 373% cpu 16,868 total
603M    /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level $i &>   77,13s user 1,76s system 307% cpu 25,655 total
603M    /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level $i &>   220,34s user 1,56s system 295% cpu 1:15,10 total
598M    /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level $i &>   290,87s user 1,82s system 334% cpu 1:27,62 total
598M    /tmp/x/x

> time mksquashfs wineprefix /tmp/x/x -b 1M -comp xz &> /dev/null; du -h /tmp/x/x; rm -f /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp xz &> /dev/null  365,15s user 2,01s system 339% cpu 1:48,14 total
589M    /tmp/x/x

> time mksquashfs wineprefix /tmp/x/x -b 1M -comp xz -Xdict-size 100% &> /dev/null; du -h /tmp/x/x; rm -f /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp xz -Xdict-size 100% &> /dev/null  366,67s user 1,84s system 342% cpu 1:47,59 total
589M    /tmp/x/x

> time mksquashfs wineprefix /tmp/x/x -b 1M -comp xz -Xdict-size 100% -Xbcj x86 &> /dev/null; du -h /tmp/x/x; rm -f /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp xz -Xdict-size 100% -Xbcj x86 &>   731,24s user 2,21s system 324% cpu 3:46,12 total
588M    /tmp/x/x

Didn't test decompression speed, as I have no doubt zstd is much better at it.

  1. this command only applies to packaging wine
  2. the prefix is not compressed at all
  3. in the choice of format, we are limited to the squashfuse utility
  4. game data is compressed by this command
    cmd = `mksquashfs "${folder}" "${squashfs}" -b 1048576 -comp lz4 -Xhc`;
  1. Ah, ok. Might be worth it to look at dwarfs if there would be multiple versions of wine and you expect data to be highly redundant.
  2. Ok, I probably got wrong impression from reading readme
  3. squashfuse seems to support zstd
  4. lz4 is nice, but zstd would probably be even better 😄
    In tests below, zstd does its job 10+ times faster while compressing better (even on compression level 1):
> du -sh wineprefix
723M    wineprefix

> time mksquashfs wineprefix /tmp/x/x -b 1M -comp lz4 -Xhc &> /dev/null; du -h /tmp/x/x; rm -f /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp lz4 -Xhc &> /dev/null  75,28s user 1,24s system 375% cpu 20,389 total
625M    /tmp/x/x

> time mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level 1 &> /dev/null; du -h /tmp/x/x; rm -f /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level 1 &>   4,54s user 1,39s system 372% cpu 1,588 total
615M    /tmp/x/x
> du -sh ~/.config
707M    /home/x/.config

> time mksquashfs ~/.config /tmp/x/x -b 1M -comp lz4 -Xhc &> /dev/null; du -h /tmp/x/x; rm -f /tmp/x/x 
mksquashfs ~/.config /tmp/x/x -b 1M -comp lz4 -Xhc &> /dev/null  77,89s user 1,35s system 361% cpu 21,943 total
541M    /tmp/x/x

> time mksquashfs ~/.config /tmp/x/x -b 1M -comp zstd -Xcompression-level 1 &> /dev/null; du -h /tmp/x/x; rm -f /tmp/x/x
mksquashfs ~/.config /tmp/x/x -b 1M -comp zstd -Xcompression-level 1 &>   5,37s user 1,40s system 353% cpu 1,914 total
537M    /tmp/x/x

1. dwarfs has an executable too large compared to squashfuse (16 MB+ vs 535 KB)
4. decompression speed is more critical than compression

  1. Might save much more, but I get your point. In this case zstd is still a good alternative to xz.
  2. Yeah, I guess... lz4hc was ~30% faster at decompression in my in-memory and SSD tests. That compression speed difference though : ) Also, zstd compression ratio is significantly better on higher compression levels. Maybe it's worth to be added as an option some day 😉