Use faster zstd compression settings

Question

Use faster zstd compression settings

Closed this issue 18 days ago · 4 comments

Currently, DKMS uses compression level 19 when ZSTD is selected as the compression method. However, this does not provide a significant benefit in terms of compressed size when compared to lower levels, especially when considering the time it takes to compress them:

As an example, here are the compression stats for nvidia-dkms on Arch Linux for compression levels -19 and -1.

with -19:

 ╭─sk@a5 in ~ 
 ╰─λ zstd -l /var/lib/dkms/nvidia/560.35.03/6.11.5-zen1-1.1-zen/x86_64/module/nvidia*.zst  
Frames  Skips  Compressed  Uncompressed  Ratio  Check  Filename
     1      0    58.1 KiB       289 KiB  4.976  XXH64  /var/lib/dkms/nvidia/560.35.03/6.11.5-zen1-1.1-zen/x86_64/module/nvidia-drm.ko.zst
     1      0    49.7 MiB      96.8 MiB  1.947  XXH64  /var/lib/dkms/nvidia/560.35.03/6.11.5-zen1-1.1-zen/x86_64/module/nvidia.ko.zst
     1      0     685 KiB      2.72 MiB  4.072  XXH64  /var/lib/dkms/nvidia/560.35.03/6.11.5-zen1-1.1-zen/x86_64/module/nvidia-modeset.ko.zst
     1      0    2.23 KiB      8.54 KiB  3.824  XXH64  /var/lib/dkms/nvidia/560.35.03/6.11.5-zen1-1.1-zen/x86_64/module/nvidia-peermem.ko.zst
     1      0     839 KiB      6.22 MiB  7.599  XXH64  /var/lib/dkms/nvidia/560.35.03/6.11.5-zen1-1.1-zen/x86_64/module/nvidia-uvm.ko.zst
----------------------------------------------------------------- 
     5      0    51.2 MiB       106 MiB  2.068  XXH64  5 files

with -1:

 ╭─sk@a5 in ~ 
 ╰─λ zstd -l /var/lib/dkms/nvidia/560.35.03/6.11.5-zen1-1.1-zen/x86_64/module/nvidia*.zst
Frames  Skips  Compressed  Uncompressed  Ratio  Check  Filename
     1      0    73.4 KiB       289 KiB  3.940  XXH64  /var/lib/dkms/nvidia/560.35.03/6.11.5-zen1-1.1-zen/x86_64/module/nvidia-drm.ko.zst
     1      0    53.5 MiB      96.8 MiB  1.807  XXH64  /var/lib/dkms/nvidia/560.35.03/6.11.5-zen1-1.1-zen/x86_64/module/nvidia.ko.zst
     1      0     825 KiB      2.72 MiB  3.383  XXH64  /var/lib/dkms/nvidia/560.35.03/6.11.5-zen1-1.1-zen/x86_64/module/nvidia-modeset.ko.zst
     1      0    2.46 KiB      8.54 KiB  3.477  XXH64  /var/lib/dkms/nvidia/560.35.03/6.11.5-zen1-1.1-zen/x86_64/module/nvidia-peermem.ko.zst
     1      0    1.05 MiB      6.22 MiB  5.954  XXH64  /var/lib/dkms/nvidia/560.35.03/6.11.5-zen1-1.1-zen/x86_64/module/nvidia-uvm.ko.zst
----------------------------------------------------------------- 
     5      0    55.5 MiB       106 MiB  1.911  XXH64  5 files

IMO, a saving of only 4.2MiB is not really worth it when the compression takes under a second for -1 but nearly 2 minutes(!) for -19.¹

This situation could be improved with either a lower compression level or by making the compression settings configurable (with some documentation of the compatibility concerns noted in #386)

Times are for my personal computer, with a threadripper 2950x @ 3.5GHz with PBO enabled, 64GB of 2933MHz ECC memory, and an nvme drive. ↩

Answer 1 · 2024-11-04T16:22:57.000Z

I think we could remove the parameter completely and just use the default (-3). Probably we can also do the same for xz and gzip.

Answer 2 · 2024-11-04T16:23:13.000Z

Care to make a merge request? Thanks.

Answer 3 · 2024-11-06T19:35:52.000Z

I think we could remove the parameter completely and just use the default (-3). Probably we can also do the same for xz and gzip.

This will lead to people with under-sized ESPs mounted on /boot suddenly not being able to run dkms due to the images (in this example, at least) taking 4MB * amount_of_kernels more.

It's arguably user-error, but the option to specify compression might still be nice to add, at least in the future.

Answer 4 · 2024-11-06T19:41:57.000Z

I think we could remove the parameter completely and just use the default (-3). Probably we can also do the same for xz and gzip.

This will lead to people with under-sized ESPs mounted on /boot suddenly not being able to run dkms due to the images taking 4MB * amount_of_kernels more.

It's arguably user-error, but the option to specify compression might still be nice to add, at least in the future.

I agree. A configuration option would be the most robust solution. IMO we should add a compression option (probably per-format) and keep the current settings as the default until the next major version to avoid breaking people's setups.

Arch's makepkg's way of handling compression settings is a pretty good example of what I'd want to see, with the caveat that the compression format would still be auto-detected: https://wiki.archlinux.org/title/Makepkg#Compression

Footnotes