Gracefuly handle available CPU cores

[blackliner]

Is your feature request related to a problem? Please describe.
As described in https://nixos.org/manual/nix/stable/advanced-topics/cores-vs-jobs, and it applies to bazel too, it can happen that you trash your system, or even go into OOM issues, when too many parallel nix derivates are being built. The goal is to reduce this number to the number of total available cores (mind: cgroups (!) in containers), to avoid this. Preferable in a dynamic way.

Describe the solution you'd like
The nix build should be executed with a limited number of available cores. Not sure if this is possible, but a good solution would also make sure that a single nix build will also "consume" multiple action slots in bazel, so bazel also "knows" about the load and does not spawn too many other things in parallel.

Describe alternatives you've considered
In our CI, we hardcode NIX_BUILD_CORES to two.

Additional context
N/A

[benradf]

Not sure if this is possible, but a good solution would also make sure that a single nix build will also "consume" multiple action slots in bazel, so bazel also "knows" about the load and does not spawn too many other things in parallel.

Agreed, this would be ideal. Not sure if you've seen it already, but there's an open Bazel issue on this topic: Cooperative Parallelism #10443.

Essentially we need Bazel to provide some kind of API that we can use to cooperatively share available cores.

[blackliner]

Thanks, yes that issue looks a) very promising and b) very stale 🤣

But bazelbuild/bazel#10443 (comment) made a good suggestion: resource_set
Can we use them to assign the nix rules one of these "resources" (lets say we have 4 of them), and then call nix with 4 cores on a 16 core machine? (TBH not 100% sure how resource_set works)