NixOS/nix

nix build --all: build all packages in a flake

Opened this issue · 17 comments

domenkozar commented

There's currently no way to build all packages in a flake, a quite common operation that was previously just nix-build.

You can see here people struggling: https://www.reddit.com/r/NixOS/comments/vohopf/is_there_an_easy_way_to_build_all_the_packages_of/

bjornfor commented

A possible workaround is to add all packages as "checks" and then "nix flake check" will build all.

  • 😕4
martyall commented

There is also a workaround using symlinkJoin and defining an all default package. You can find an example of this in this article in the section titled "The Flake"

hsjobeki commented

just ran into the same issue.
what if we could just add the --all feature to flakes which builds all packages as expected?

  • 👍2
carlthome commented

just ran into the same issue.
what if we could just add the --all feature to flakes which builds all packages as expected?

That's what I assumed already existed, at least.

Or, when no default package is set, that "nix build" would build everything.

bobvanderlinden commented

What should be the output of --all? For instance, should it result in a bunch of result-{1,2,3,4,5} symlinks?

hsjobeki commented

What should be the output of --all? For instance, should it result in a bunch of result-{1,2,3,4,5} symlinks?

Good question. I guess this would clutter your workspace on larger package sets.
I‘d like a folder in the nix store which holds a list of symlinks to every single package

bobvanderlinden commented

So it'll basically implicitly create a new derivation using --all?

linkFarm [
  { name = "packagea"; path = packages.${system}.packagea }
  { name = "packageb"; path = packages.${system}.packageb }
  { name = "packageb"; path = packages.${system}.packageb }
]

Resulting in a result symlink:

result/packagea -> /nix/store/...-packagea/
result/packageb -> /nix/store/...-packageb/
result/packagec -> /nix/store/...-packagec/

I'm not sure I like the approach, as it is non-transparent if Nix is going to build derivations implicitly.

It could also be a convention or even a default to make packages.${system}.default or packages.${system}.all use the linkFarm approach. If it was a convention, a helper function that creates a 'package set' derivation would be useful.

Then it would work without needing a --all option.

aciceri commented

I don't like it, shouldn't have to depend on nixpkgs in my opinion. Moreover derivations can have multiple outputs (can't remember how nix build manages it currently, not even sure it does).

  • 👍1
bobvanderlinden commented

What do you think is the better approach to solve the issue?

shouldn't have to depend on nixpkgs in my opinion

There could also be a builtin for creating a 'package set', but I would agree is it's not ideal.

Moreover derivations can have multiple outputs

Indeed, that's why I thought earlier that it would output result-{1,2,3,4,5}.

Would it output result-{packagea,packageb,packagec} instead?

aciceri commented

I meant multiple outputs per single derivation, so something like result-{drvName}-{outputName}.

  • 👍2
iFreilicht commented

This seems to be a duplicate of #3811, though I would vote to close the older issue, as there is more discussion in this thread here.

The gist of #3811 is that the behavior of nix build --all is not intuitively well-defined, as a flake can have any number of outputs, of which only some are known.

For example, the nix repo itself has all these outputs:

There's a lot, click here to expand... 
$ nix flake show
git+file:///Users/feuh/repos/nix
├───checks
│   ├───aarch64-darwin
│   │   ├───binaryTarball: derivation 'nix-binary-tarball-2.21.0pre20240305_dirty'
│   │   ├───installTests: derivation 'install-tests'
│   │   ├───nixpkgsLibTests: derivation 'nixpkgs-lib-tests'
│   │   ├───perlBindings: derivation 'nix-perl-2.21.0pre20240305_dirty'
│   │   └───rl-next: derivation 'test-rl-next-release-notes'
│   ├───aarch64-linux
│   │   ├───binaryTarball omitted (use '--all-systems' to show)
│   │   ├───dockerImage omitted (use '--all-systems' to show)
│   │   ├───installTests omitted (use '--all-systems' to show)
│   │   ├───nixpkgsLibTests omitted (use '--all-systems' to show)
│   │   ├───perlBindings omitted (use '--all-systems' to show)
│   │   └───rl-next omitted (use '--all-systems' to show)
│   ├───i686-linux
│   │   ├───binaryTarball omitted (use '--all-systems' to show)
│   │   ├───installTests omitted (use '--all-systems' to show)
│   │   ├───nixpkgsLibTests omitted (use '--all-systems' to show)
│   │   ├───perlBindings omitted (use '--all-systems' to show)
│   │   └───rl-next omitted (use '--all-systems' to show)
│   ├───x86_64-darwin
│   │   ├───binaryTarball omitted (use '--all-systems' to show)
│   │   ├───installTests omitted (use '--all-systems' to show)
│   │   ├───nixpkgsLibTests omitted (use '--all-systems' to show)
│   │   ├───perlBindings omitted (use '--all-systems' to show)
│   │   └───rl-next omitted (use '--all-systems' to show)
│   └───x86_64-linux
│       ├───binaryTarball omitted (use '--all-systems' to show)
│       ├───dockerImage omitted (use '--all-systems' to show)
│       ├───installTests omitted (use '--all-systems' to show)
│       ├───nixpkgsLibTests omitted (use '--all-systems' to show)
│       ├───perlBindings omitted (use '--all-systems' to show)
│       └───rl-next omitted (use '--all-systems' to show)
├───devShells
│   ├───aarch64-darwin
│   │   ├───armv6l-unknown-linux-gnueabihf: development environment 'nix-armv6l-unknown-linux-gnueabihf-2.21.0pre20240305_dirty'
│   │   ├───armv7l-unknown-linux-gnueabihf: development environment 'nix-armv7l-unknown-linux-gnueabihf-2.21.0pre20240305_dirty'
│   │   ├───default: development environment 'nix-2.21.0pre20240305_dirty'
│   │   ├───native-ccacheStdenvPackages: development environment 'nix-2.21.0pre20240305_dirty'
│   │   ├───native-clangStdenvPackages: development environment 'nix-2.21.0pre20240305_dirty'
│   │   ├───native-gccStdenvPackages: development environment 'nix-2.21.0pre20240305_dirty'
│   │   ├───native-libcxxStdenvPackages: development environment 'nix-2.21.0pre20240305_dirty'
│   │   ├───native-stdenvPackages: development environment 'nix-2.21.0pre20240305_dirty'
│   │   ├───x86_64-unknown-freebsd13: development environment 'nix-x86_64-unknown-freebsd13-2.21.0pre20240305_dirty'
│   │   ├───x86_64-unknown-netbsd: development environment 'nix-x86_64-unknown-netbsd-2.21.0pre20240305_dirty'
│   │   └───x86_64-w64-mingw32: development environment 'nix-x86_64-w64-mingw32-2.21.0pre20240305_dirty'
│   ├───aarch64-linux
│   │   ├───armv6l-unknown-linux-gnueabihf omitted (use '--all-systems' to show)
│   │   ├───armv7l-unknown-linux-gnueabihf omitted (use '--all-systems' to show)
│   │   ├───default omitted (use '--all-systems' to show)
│   │   ├───native-ccacheStdenvPackages omitted (use '--all-systems' to show)
│   │   ├───native-clangStdenvPackages omitted (use '--all-systems' to show)
│   │   ├───native-gccStdenvPackages omitted (use '--all-systems' to show)
│   │   ├───native-libcxxStdenvPackages omitted (use '--all-systems' to show)
│   │   ├───native-stdenvPackages omitted (use '--all-systems' to show)
│   │   ├───static-ccacheStdenvPackages omitted (use '--all-systems' to show)
│   │   ├───static-clangStdenvPackages omitted (use '--all-systems' to show)
│   │   ├───static-gccStdenvPackages omitted (use '--all-systems' to show)
│   │   ├───static-libcxxStdenvPackages omitted (use '--all-systems' to show)
│   │   ├───static-stdenvPackages omitted (use '--all-systems' to show)
│   │   ├───x86_64-unknown-freebsd13 omitted (use '--all-systems' to show)
│   │   ├───x86_64-unknown-netbsd omitted (use '--all-systems' to show)
│   │   └───x86_64-w64-mingw32 omitted (use '--all-systems' to show)
│   ├───i686-linux
│   │   ├───armv6l-unknown-linux-gnueabihf omitted (use '--all-systems' to show)
│   │   ├───armv7l-unknown-linux-gnueabihf omitted (use '--all-systems' to show)
│   │   ├───default omitted (use '--all-systems' to show)
│   │   ├───native-ccacheStdenvPackages omitted (use '--all-systems' to show)
│   │   ├───native-clangStdenvPackages omitted (use '--all-systems' to show)
│   │   ├───native-gccStdenvPackages omitted (use '--all-systems' to show)
│   │   ├───native-libcxxStdenvPackages omitted (use '--all-systems' to show)
│   │   ├───native-stdenvPackages omitted (use '--all-systems' to show)
│   │   ├───static-ccacheStdenvPackages omitted (use '--all-systems' to show)
│   │   ├───static-clangStdenvPackages omitted (use '--all-systems' to show)
│   │   ├───static-gccStdenvPackages omitted (use '--all-systems' to show)
│   │   ├───static-libcxxStdenvPackages omitted (use '--all-systems' to show)
│   │   ├───static-stdenvPackages omitted (use '--all-systems' to show)
│   │   ├───x86_64-unknown-freebsd13 omitted (use '--all-systems' to show)
│   │   ├───x86_64-unknown-netbsd omitted (use '--all-systems' to show)
│   │   └───x86_64-w64-mingw32 omitted (use '--all-systems' to show)
│   ├───x86_64-darwin
│   │   ├───armv6l-unknown-linux-gnueabihf omitted (use '--all-systems' to show)
│   │   ├───armv7l-unknown-linux-gnueabihf omitted (use '--all-systems' to show)
│   │   ├───default omitted (use '--all-systems' to show)
│   │   ├───native-ccacheStdenvPackages omitted (use '--all-systems' to show)
│   │   ├───native-clangStdenvPackages omitted (use '--all-systems' to show)
│   │   ├───native-gccStdenvPackages omitted (use '--all-systems' to show)
│   │   ├───native-libcxxStdenvPackages omitted (use '--all-systems' to show)
│   │   ├───native-stdenvPackages omitted (use '--all-systems' to show)
│   │   ├───x86_64-unknown-freebsd13 omitted (use '--all-systems' to show)
│   │   ├───x86_64-unknown-netbsd omitted (use '--all-systems' to show)
│   │   └───x86_64-w64-mingw32 omitted (use '--all-systems' to show)
│   └───x86_64-linux
│       ├───armv6l-unknown-linux-gnueabihf omitted (use '--all-systems' to show)
│       ├───armv7l-unknown-linux-gnueabihf omitted (use '--all-systems' to show)
│       ├───default omitted (use '--all-systems' to show)
│       ├───native-ccacheStdenvPackages omitted (use '--all-systems' to show)
│       ├───native-clangStdenvPackages omitted (use '--all-systems' to show)
│       ├───native-gccStdenvPackages omitted (use '--all-systems' to show)
│       ├───native-libcxxStdenvPackages omitted (use '--all-systems' to show)
│       ├───native-stdenvPackages omitted (use '--all-systems' to show)
│       ├───static-ccacheStdenvPackages omitted (use '--all-systems' to show)
│       ├───static-clangStdenvPackages omitted (use '--all-systems' to show)
│       ├───static-gccStdenvPackages omitted (use '--all-systems' to show)
│       ├───static-libcxxStdenvPackages omitted (use '--all-systems' to show)
│       ├───static-stdenvPackages omitted (use '--all-systems' to show)
│       ├───x86_64-unknown-freebsd13 omitted (use '--all-systems' to show)
│       ├───x86_64-unknown-netbsd omitted (use '--all-systems' to show)
│       └───x86_64-w64-mingw32 omitted (use '--all-systems' to show)
├───hydraJobs
│   ├───binaryTarball
│   │   ├───aarch64-darwin: derivation 'nix-binary-tarball-2.21.0pre20240305_dirty'
│   │   ├───aarch64-linux: derivation 'nix-binary-tarball-2.21.0pre20240305_dirty'
│   │   ├───i686-linux: derivation 'nix-binary-tarball-2.21.0pre20240305_dirty'
│   │   ├───x86_64-darwin: derivation 'nix-binary-tarball-2.21.0pre20240305_dirty'
│   │   └───x86_64-linux: derivation 'nix-binary-tarball-2.21.0pre20240305_dirty'
│   ├───binaryTarballCross
│   │   └───x86_64-linux
│   │       ├───armv6l-unknown-linux-gnueabihf: derivation 'nix-binary-tarball-2.21.0pre20240305_dirty'
│   │       ├───armv7l-unknown-linux-gnueabihf: derivation 'nix-binary-tarball-2.21.0pre20240305_dirty'
│   │       ├───x86_64-unknown-freebsd13: derivation 'nix-binary-tarball-2.21.0pre20240305_dirty'
│   │       └───x86_64-unknown-netbsd: derivation 'nix-binary-tarball-2.21.0pre20240305_dirty'
│   ├───build
│   │   ├───aarch64-darwin: derivation 'nix-2.21.0pre20240305_dirty'
│   │   ├───aarch64-linux: derivation 'nix-2.21.0pre20240305_dirty'
│   │   ├───i686-linux: derivation 'nix-2.21.0pre20240305_dirty'
│   │   ├───x86_64-darwin: derivation 'nix-2.21.0pre20240305_dirty'
│   │   └───x86_64-linux: derivation 'nix-2.21.0pre20240305_dirty'
│   ├───buildCross
│   │   ├───armv6l-unknown-linux-gnueabihf
│   │   │   └───x86_64-linux: derivation 'nix-armv6l-unknown-linux-gnueabihf-2.21.0pre20240305_dirty'
│   │   ├───armv7l-unknown-linux-gnueabihf
│   │   │   └───x86_64-linux: derivation 'nix-armv7l-unknown-linux-gnueabihf-2.21.0pre20240305_dirty'
│   │   ├───x86_64-unknown-freebsd13
│   │   │   └───x86_64-linux: derivation 'nix-x86_64-unknown-freebsd13-2.21.0pre20240305_dirty'
│   │   └───x86_64-unknown-netbsd
│   │       └───x86_64-linux: derivation 'nix-x86_64-unknown-netbsd-2.21.0pre20240305_dirty'
│   ├───buildNoGc
│   │   ├───aarch64-darwin: derivation 'nix-2.21.0pre20240305_dirty'
│   │   ├───aarch64-linux: derivation 'nix-2.21.0pre20240305_dirty'
│   │   ├───i686-linux: derivation 'nix-2.21.0pre20240305_dirty'
│   │   ├───x86_64-darwin: derivation 'nix-2.21.0pre20240305_dirty'
│   │   └───x86_64-linux: derivation 'nix-2.21.0pre20240305_dirty'
│   ├───buildNoTests
│   │   ├───aarch64-darwin: derivation 'nix-2.21.0pre20240305_dirty'
│   │   ├───aarch64-linux: derivation 'nix-2.21.0pre20240305_dirty'
│   │   ├───i686-linux: derivation 'nix-2.21.0pre20240305_dirty'
│   │   ├───x86_64-darwin: derivation 'nix-2.21.0pre20240305_dirty'
│   │   └───x86_64-linux: derivation 'nix-2.21.0pre20240305_dirty'
│   ├───buildReadlineNoMarkdown
│   │   ├───aarch64-darwin: derivation 'nix-2.21.0pre20240305_dirty'
│   │   ├───aarch64-linux: derivation 'nix-2.21.0pre20240305_dirty'
│   │   ├───i686-linux: derivation 'nix-2.21.0pre20240305_dirty'
│   │   ├───x86_64-darwin: derivation 'nix-2.21.0pre20240305_dirty'
│   │   └───x86_64-linux: derivation 'nix-2.21.0pre20240305_dirty'
│   ├───buildStatic
│   │   ├───aarch64-linux: derivation 'nix-static-aarch64-unknown-linux-musl-2.21.0pre20240305_dirty'
│   │   └───x86_64-linux: derivation 'nix-static-x86_64-unknown-linux-musl-2.21.0pre20240305_dirty'
│   ├───coverage: derivation 'nix-coverage'
│   ├───dockerImage
│   │   ├───aarch64-linux: derivation 'docker-image-tarball-2.21.0pre20240305_dirty'
│   │   └───x86_64-linux: derivation 'docker-image-tarball-2.21.0pre20240305_dirty'
│   ├───installTests
│   │   ├───aarch64-darwin: derivation 'install-tests'
│   │   ├───aarch64-linux: derivation 'install-tests'
│   │   ├───i686-linux: derivation 'install-tests'
│   │   ├───x86_64-darwin: derivation 'install-tests'
│   │   └───x86_64-linux: derivation 'install-tests'
│   ├───installerScript: derivation 'installer-script'
│   ├───installerScriptForGHA: derivation 'installer-script'
│   ├───installerTests
│   │   ├───fedora-36
│   │   │   └───x86_64-linux
│   │   │       ├───install-default: derivation 'installer-test-fedora-36-install-default'
│   │   │       ├───install-force-daemon: derivation 'installer-test-fedora-36-install-force-daemon'
│   │   │       └───install-force-no-daemon: derivation 'installer-test-fedora-36-install-force-no-daemon'
│   │   ├───rhel-7
│   │   │   └───x86_64-linux
│   │   │       ├───install-default: derivation 'installer-test-rhel-7-install-default'
│   │   │       ├───install-force-daemon: derivation 'installer-test-rhel-7-install-force-daemon'
│   │   │       └───install-force-no-daemon: derivation 'installer-test-rhel-7-install-force-no-daemon'
│   │   ├───rhel-8
│   │   │   └───x86_64-linux
│   │   │       ├───install-default: derivation 'installer-test-rhel-8-install-default'
│   │   │       ├───install-force-daemon: derivation 'installer-test-rhel-8-install-force-daemon'
│   │   │       └───install-force-no-daemon: derivation 'installer-test-rhel-8-install-force-no-daemon'
│   │   ├───rhel-9
│   │   │   └───x86_64-linux
│   │   │       ├───install-default: derivation 'installer-test-rhel-9-install-default'
│   │   │       ├───install-force-daemon: derivation 'installer-test-rhel-9-install-force-daemon'
│   │   │       └───install-force-no-daemon: derivation 'installer-test-rhel-9-install-force-no-daemon'
│   │   ├───ubuntu-16-04
│   │   │   └───x86_64-linux
│   │   │       ├───install-default: derivation 'installer-test-ubuntu-16-04-install-default'
│   │   │       ├───install-force-daemon: derivation 'installer-test-ubuntu-16-04-install-force-daemon'
│   │   │       └───install-force-no-daemon: derivation 'installer-test-ubuntu-16-04-install-force-no-daemon'
│   │   └───ubuntu-22-04
│   │       └───x86_64-linux
│   │           ├───install-default: derivation 'installer-test-ubuntu-22-04-install-default'
│   │           ├───install-force-daemon: derivation 'installer-test-ubuntu-22-04-install-force-daemon'
│   │           └───install-force-no-daemon: derivation 'installer-test-ubuntu-22-04-install-force-no-daemon'
│   ├───internal-api-docs: derivation 'nix-2.21.0'
│   ├───metrics
│   │   └───nixpkgs: derivation 'nixpkgs-metrics'
│   ├───perlBindings
│   │   ├───aarch64-darwin: derivation 'nix-perl-2.21.0pre20240305_dirty'
│   │   ├───aarch64-linux: derivation 'nix-perl-2.21.0pre20240305_dirty'
│   │   ├───i686-linux: derivation 'nix-perl-2.21.0pre20240305_dirty'
│   │   ├───x86_64-darwin: derivation 'nix-perl-2.21.0pre20240305_dirty'
│   │   └───x86_64-linux: derivation 'nix-perl-2.21.0pre20240305_dirty'
│   ├───shellInputs
│   │   ├───aarch64-darwin: derivation 'nix-2.21.0pre20240305_dirty'
│   │   ├───aarch64-linux: derivation 'nix-2.21.0pre20240305_dirty'
│   │   ├───i686-linux: derivation 'nix-2.21.0pre20240305_dirty'
│   │   ├───x86_64-darwin: derivation 'nix-2.21.0pre20240305_dirty'
│   │   └───x86_64-linux: derivation 'nix-2.21.0pre20240305_dirty'
│   └───tests
│       ├───authorization: derivation 'vm-test-run-authorization'
│       ├───containers: derivation 'vm-test-run-containers'
│       ├───evalNixpkgs: derivation 'eval-nixos'
│       ├───fetch-git: derivation 'vm-test-run-fetch-git'
│       ├───githubFlakes: derivation 'vm-test-run-github-flakes'
│       ├───nix-copy: derivation 'vm-test-run-nix-copy'
│       ├───nix-copy-closure: derivation 'vm-test-run-nix-copy-closure'
│       ├───nixpkgsLibTests
│       │   ├───aarch64-darwin: derivation 'nixpkgs-lib-tests'
│       │   ├───aarch64-linux: derivation 'nixpkgs-lib-tests'
│       │   ├───i686-linux: derivation 'nixpkgs-lib-tests'
│       │   ├───x86_64-darwin: derivation 'nixpkgs-lib-tests'
│       │   └───x86_64-linux: derivation 'nixpkgs-lib-tests'
│       ├───nssPreload: derivation 'vm-test-run-nss-preload'
│       ├───remoteBuilds: derivation 'vm-test-run-remote-builds'
│       ├───remoteBuildsSshNg: derivation 'vm-test-run-remote-builds-ssh-ng'
│       ├───remoteBuildsSshNg_remote_2_13: derivation 'vm-test-run-remoteBuildsSshNg_remote_2_13'
│       ├───remoteBuildsSshNg_remote_2_3: derivation 'vm-test-run-remoteBuildsSshNg_remote_2_3'
│       ├───remoteBuilds_local_2_13: derivation 'vm-test-run-remoteBuilds_local_2_13'
│       ├───remoteBuilds_local_2_3: derivation 'vm-test-run-remoteBuilds_local_2_3'
│       ├───remoteBuilds_remote_2_13: derivation 'vm-test-run-remoteBuilds_remote_2_13'
│       ├───remoteBuilds_remote_2_3: derivation 'vm-test-run-remoteBuilds_remote_2_3'
│       ├───setuid
│       │   ├───i686-linux: derivation 'vm-test-run-setuid'
│       │   └───x86_64-linux: derivation 'vm-test-run-setuid'
│       ├───sourcehutFlakes: derivation 'vm-test-run-sourcehut-flakes'
│       └───tarballFlakes: derivation 'vm-test-run-tarball-flakes'
├───overlays
│   └───default: Nixpkgs overlay
└───packages
    ├───aarch64-darwin
    │   ├───changelog-d-nix: package 'changelog-d-0.1-git-f30f696'
    │   ├───default: package 'nix-2.21.0pre20240305_dirty'
    │   ├───nix: package 'nix-2.21.0pre20240305_dirty'
    │   ├───nix-armv6l-unknown-linux-gnueabihf: package 'nix-armv6l-unknown-linux-gnueabihf-2.21.0pre20240305_dirty'
    │   ├───nix-armv7l-unknown-linux-gnueabihf: package 'nix-armv7l-unknown-linux-gnueabihf-2.21.0pre20240305_dirty'
    │   ├───nix-ccacheStdenv: package 'nix-2.21.0pre20240305_dirty'
    │   ├───nix-clangStdenv: package 'nix-2.21.0pre20240305_dirty'
    │   ├───nix-gccStdenv: package 'nix-2.21.0pre20240305_dirty'
    │   ├───nix-libcxxStdenv: package 'nix-2.21.0pre20240305_dirty'
    │   ├───nix-stdenv: package 'nix-2.21.0pre20240305_dirty'
    │   ├───nix-x86_64-unknown-freebsd13: package 'nix-x86_64-unknown-freebsd13-2.21.0pre20240305_dirty'
    │   └───nix-x86_64-unknown-netbsd: package 'nix-x86_64-unknown-netbsd-2.21.0pre20240305_dirty'
    ├───aarch64-linux
    │   ├───changelog-d-nix omitted (use '--all-systems' to show)
    │   ├───default omitted (use '--all-systems' to show)
    │   ├───dockerImage omitted (use '--all-systems' to show)
    │   ├───nix omitted (use '--all-systems' to show)
    │   ├───nix-armv6l-unknown-linux-gnueabihf omitted (use '--all-systems' to show)
    │   ├───nix-armv7l-unknown-linux-gnueabihf omitted (use '--all-systems' to show)
    │   ├───nix-ccacheStdenv omitted (use '--all-systems' to show)
    │   ├───nix-clangStdenv omitted (use '--all-systems' to show)
    │   ├───nix-gccStdenv omitted (use '--all-systems' to show)
    │   ├───nix-libcxxStdenv omitted (use '--all-systems' to show)
    │   ├───nix-static omitted (use '--all-systems' to show)
    │   ├───nix-stdenv omitted (use '--all-systems' to show)
    │   ├───nix-x86_64-unknown-freebsd13 omitted (use '--all-systems' to show)
    │   └───nix-x86_64-unknown-netbsd omitted (use '--all-systems' to show)
    ├───i686-linux
    │   ├───changelog-d-nix omitted (use '--all-systems' to show)
    │   ├───default omitted (use '--all-systems' to show)
    │   ├───nix omitted (use '--all-systems' to show)
    │   ├───nix-armv6l-unknown-linux-gnueabihf omitted (use '--all-systems' to show)
    │   ├───nix-armv7l-unknown-linux-gnueabihf omitted (use '--all-systems' to show)
    │   ├───nix-ccacheStdenv omitted (use '--all-systems' to show)
    │   ├───nix-clangStdenv omitted (use '--all-systems' to show)
    │   ├───nix-gccStdenv omitted (use '--all-systems' to show)
    │   ├───nix-libcxxStdenv omitted (use '--all-systems' to show)
    │   ├───nix-stdenv omitted (use '--all-systems' to show)
    │   ├───nix-x86_64-unknown-freebsd13 omitted (use '--all-systems' to show)
    │   └───nix-x86_64-unknown-netbsd omitted (use '--all-systems' to show)
    ├───x86_64-darwin
    │   ├───changelog-d-nix omitted (use '--all-systems' to show)
    │   ├───default omitted (use '--all-systems' to show)
    │   ├───nix omitted (use '--all-systems' to show)
    │   ├───nix-armv6l-unknown-linux-gnueabihf omitted (use '--all-systems' to show)
    │   ├───nix-armv7l-unknown-linux-gnueabihf omitted (use '--all-systems' to show)
    │   ├───nix-ccacheStdenv omitted (use '--all-systems' to show)
    │   ├───nix-clangStdenv omitted (use '--all-systems' to show)
    │   ├───nix-gccStdenv omitted (use '--all-systems' to show)
    │   ├───nix-libcxxStdenv omitted (use '--all-systems' to show)
    │   ├───nix-stdenv omitted (use '--all-systems' to show)
    │   ├───nix-x86_64-unknown-freebsd13 omitted (use '--all-systems' to show)
    │   └───nix-x86_64-unknown-netbsd omitted (use '--all-systems' to show)
    └───x86_64-linux
        ├───changelog-d-nix omitted (use '--all-systems' to show)
        ├───default omitted (use '--all-systems' to show)
        ├───dockerImage omitted (use '--all-systems' to show)
        ├───nix omitted (use '--all-systems' to show)
        ├───nix-armv6l-unknown-linux-gnueabihf omitted (use '--all-systems' to show)
        ├───nix-armv7l-unknown-linux-gnueabihf omitted (use '--all-systems' to show)
        ├───nix-ccacheStdenv omitted (use '--all-systems' to show)
        ├───nix-clangStdenv omitted (use '--all-systems' to show)
        ├───nix-gccStdenv omitted (use '--all-systems' to show)
        ├───nix-libcxxStdenv omitted (use '--all-systems' to show)
        ├───nix-static omitted (use '--all-systems' to show)
        ├───nix-stdenv omitted (use '--all-systems' to show)
        ├───nix-x86_64-unknown-freebsd13 omitted (use '--all-systems' to show)
        └───nix-x86_64-unknown-netbsd omitted (use '--all-systems' to show)

As a summary, these are the categories:

  • checks
  • devShells
  • hydraJobs
  • overlays
  • packages

Many of them are available for five platforms, but some aren't. What should be built when running nix build --all? Would everyone have the same expectation?

As I understand the sentiment of this thread, the general expectation would be for it to build all packages for the current architecture. Potentially, adding the same --all-systems flag that nix flake show already has might be a reasonable addition, as well as allowing --all to be passed multiple times with an optional value that allows building all devShells or any other set of outputs.

$ nix build --all  # Builds all packages
$ nix build --all packages   # Same as above
$ nix build --all packages --all devShells # Build all packages and devShells
$ nix build --all devShells --all-systems # Build all devShells for all systems (requires remote builders)
$ nix build --all --system x86_64-linux --system aarch64-linux # Another flag for selecting specific systems

I think --all-systems would be very niche, though. I guess I can see the use when doing release builds and you indeed have remote builders for all the systems.

A (neat but complex) alternative would be to add more syntax to flakerefs:

$ nix build '.#*'  # * could be any other character, but has precedent

$ nix build '.#packages.aarch64-darwin.*' # Have to type out system tuple
$ nix build '.#packages.$system.*'  # or add special case
$ nix build ".#packages.$(nix system).*" # or additional command and substitute in the shell

$ nix build .#{packages,devShells}.aarch64-darwin."*" # Shell expansion allows more freedom, but also counterintuitive

$ nix build ".#devShells.*.*" # The most niche usecase is the shortest

$ nix build .#packages.{x86_64,aarch64}-{linux,darwin}."*"  # And you can do clever stuff like this

I'd say the latter approach feels very powerful and composable, but the first one is definitely much easier to discover, and probably covers most use-cases already. They don't have to be mutually exclusive, but more features isn't necessarily better.

Another way of solving this could be with nix code, if #5567 ever gets implemented, though that would be far from an intuitive alternative to --all, more of an argument against implementing a syntax extension to flakerefs.

Proposal

I would suggest to start a first implementation with the following behavior:

$ nix build --all
error: --all requires the name of a package set.
       The following package sets are available in the selected flake:
         checks
         packages
         devShells
$ nix build --all packages

For starters, the outputs would be numbered results, that is in line with the current behavior of nix build when multiple derivations are built.

This is obviously a limited and minimally viable solution, but it avoids ambiguity and is relatively easy to implement, which means the feature can be added and maintained with limited effort. It is also easy to extend in the future without breaking backwards compatibility.

  • 👍9
siriobalmelli commented

In case it helps, I've made this shell function to build and push all flake inputs and outputs for myself:

cashout() {
	set -x
	nix flake archive --json |
		jq -r '.path,(.inputs|to_entries[].value.path)' |
		gopass env "env/$1/cachix_auth_token" cachix push "$1"
	for target in $(
		nix flake show --json --all-systems | jq '
		["checks", "packages", "devShells"] as $tops | 
		$tops[] as $top |
		.[$top] | 
		to_entries[] | 
		.key as $arch | 
		.value | 
		keys[] | 
		"\($top).\($arch).\(.)"
		' | tr -d '"'
	); do
		nix build --json ".#$target" "${@:2}" |
			jq -r '.[].outputs | to_entries[].value' |
			gopass env "env/$1/cachix_auth_token" cachix push "$1"
	done
}
fzakaria commented

what about recurseIntoAttrs ?

siriobalmelli commented

I'm embarrassed to say I don't fully understand how recurseIntoAttrs would be used in this case. Any example would be helpful.

hsjobeki commented

This is an issue with the new nix commands. Thats also why this issue is open in the nix repository and not nixpkgs.

recurseIntoAttrs acts as hand-maintained guide for some tools in nix, where to recurse for automatically discovering packages/derivations. I am not long enough into nix but i think recurseIntoAttrs was invented because it is impossble to use nix itself to recursively evaluate nixpkgs and find all potentially deeply nested packages. So this function evolved into a convention and adds little hints to the expression tree, where to stop/continue with the recursion.

Now this convention was respected by nix itself:

nix/src/libexpr/get-drvs.cc

Line 386 in e943ee3

`recurseForDerivations = true' attribute. */

But the problem is that the new command nix build doesn't respect it as before via nix-build.

(Reference to the new build command:

nix/src/nix/build.cc

Line 116 in e943ee3

void run(ref<Store> store, Installables && installables) override
)

  • 👀1