build.disk-image via binfmt taking 40mins doing a find/du on my /nix/store

[tomfitzhenry]

nixpkgs: 7e567a3d092b7de69cdf5deaeb8d9526de230916
mobile-nixos: 85557dc

(Both taken from https://hydra.nixos.org/build/146100953#tabs-buildinputs )

I'm running a NixOS 21.05 x86-64 machine with aarch64-linux binfmt enabled, with btrfs on an NVMe drive.

Here's my local.nix:

{ lib, pkgs, ... }:

{
  imports = [
    ./examples/demo/configuration.nix
  ];
  nixpkgs.system = "aarch64-linux";
}

I executed -A build.disk-image, which pulled a bunch from cache.nixos.org and then has spent 28 minutes stuck on:

Done copying system closure...
'/nix/store/w945b9mikl1c480xnkx3w9lhc8pi28bl-closure-info/registration' -> './nix-path-registration'

Full command output: https://paste.debian.net/1203023/

Some diagnostics

pstree shows nix-daemon is building something that triggers find with du:

 | \-+= 43478 root nix-daemon 43465    
 |   |-+= 47223 nixbld1 /nix/store/4s5s0bgp6708nnyl9zbc7fa6s8c5xh59-qemu-6.0.0/bin/qemu-aarch64 /nix/store/kawx6qqji9c8jal32d554whblbjj81l3-bash-4.4-p23/bin/bash -e /nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh 
 |   | \-+- 50100 nixbld1 /nix/store/4s5s0bgp6708nnyl9zbc7fa6s8c5xh59-qemu-6.0.0/bin/qemu-aarch64 /nix/store/kawx6qqji9c8jal32d554whblbjj81l3-bash-4.4-p23/bin/bash -e /nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh 
 |   |   \-+- 50102 nixbld1 /nix/store/4s5s0bgp6708nnyl9zbc7fa6s8c5xh59-qemu-6.0.0/bin/qemu-aarch64 /nix/store/kawx6qqji9c8jal32d554whblbjj81l3-bash-4.4-p23/bin/bash -e /nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh 
 |   |     \-+- 50104 nixbld1 /nix/store/4s5s0bgp6708nnyl9zbc7fa6s8c5xh59-qemu-6.0.0/bin/qemu-aarch64 /nix/store/kawx6qqji9c8jal32d554whblbjj81l3-bash-4.4-p23/bin/bash -e /nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh 
 |   |       |--- 50110 nixbld1 /nix/store/4s5s0bgp6708nnyl9zbc7fa6s8c5xh59-qemu-6.0.0/bin/qemu-aarch64 /nix/store/kawx6qqji9c8jal32d554whblbjj81l3-bash-4.4-p23/bin/bash -e /nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh 
 |   |       |--- 50108 nixbld1 /nix/store/4s5s0bgp6708nnyl9zbc7fa6s8c5xh59-qemu-6.0.0/bin/qemu-aarch64 /nix/store/pcc3dbxii97a919fm0bs39mqfwlzkdma-coreutils-8.32/bin/cut -f1 
 |   |       \-+- 50106 nixbld1 /nix/store/4s5s0bgp6708nnyl9zbc7fa6s8c5xh59-qemu-6.0.0/bin/qemu-aarch64 /nix/store/ns6p3l0gvl1yq4mb7a04zd2hs05sws0h-findutils-4.7.0/bin/find . ! -type d -exec du --apparent-size --block-size 4096 {} ; 
 |   |         \--- 388354 nixbld1 /nix/store/4s5s0bgp6708nnyl9zbc7fa6s8c5xh59-qemu-6.0.0/bin/qemu-aarch64 /nix/store/pcc3dbxii97a919fm0bs39mqfwlzkdma-coreutils-8.32/bin/du --apparent-size --block-size 4096 ./nix/store/9pakc0y95f0nyr0vv3x49r6b217qnd74-libfido2-1.7.0/share/man/man3/fido_assert_set_hmac_secret.3.gz 

That find has been running for 37 mins (update it took 40 mins in total), and is still running:

$ ps aux | grep 50106
nixbld1    50106  0.9  0.0 240320 15460 ?        Sl   21:48   0:20 /nix/store/4s5s0bgp6708nnyl9zbc7fa6s8c5xh59-qemu-6.0.0/bin/qemu-aarch64 /nix/store/ns6p3l0gvl1yq4mb7a04zd2hs05sws0h-findutils-4.7.0/bin/find . ! -type d -exec du --apparent-size --block-size 4096 {} ;
tom       397543  0.0  0.0 223452  2328 pts/1    S+   22:25   0:00 grep 50106

$ ps -o etime= -p 50106
      37:22

$ date
Thu Jul  1 10:25:49 PM AEST 2021

[tomfitzhenry]

After 40 mins, the build succeeded, with a 4GB disk image.

[samueldr]

find/du used to get a proper reading of the actual block size the files will require. That is, a file that is 1 byte long still take a full filesystem "block".

Otherwise, the way blocks are always rounded up could result in the filesystem being too small! Imagine 100 000 files in the filesystem, all of them being under half a 512 bytes block rounded up, so:

(256 * 100_000) / 1024.0 / 1024.0
=> 24.4140625

This is about a 24 MiB difference! The image building infra is used in a way the filesystem images are reduced to their most minimal size.

Though note that I'll be taking a look at genimage in the future. Hopefully it can help somewhat here in the binfmt use case. (I'm using the same infra to build images no that other project.)