Deployment with nixops starts 32 build processes
Closed this issue · 4 comments
I'm trying to deploy nixos VPS from another VPS with nixops. I run multiple times into an OOM killer. The build tool spawns a few dozens of processes, be it rustc (ripgrep) or webpack (gitlab assets) just before it's shut down. I managed to compile ripgrep by assigning more resources (2 GB -> 4 GB), webpack sitll fails.
The command that fails:
/tmp/nix-build-gitlab-assets-13.0.9.drv-1/source/node_modules/.bin/webpack \
--config /tmp/nix-build-gitlab-assets-13.0.9.drv-1/source/config/webpack.config.js --bail
Deploy command:
nixops deploy -d simple --max-jobs 1 --cores 1
VPS #16200 has 4 GB, it runs on node15.prg. dmesg show there were 32 node
processes when OOM killer killed webpack.
32 seems supsicious, I think the build tools get information that they should start 32 instances (maybe from host machine?). They do and OOM killer is spawned soon because there isn't enough memory on VM.
I was able to reproduce it multiple times, always with the same result.
Update 1: manual compilation passes: (it''s not full compilation, probably some env is missing)
cd /tmp/nix-build-gitlab-assets-13.0.9.drv-2/source
nix-shell /nix/store/rssbqrlxwp6ds15lnz6jww6xvr5vabp8-gitlab-assets-13.0.9.drv
tmp/nix-build-gitlab-assets-13.0.9.drv-2/source/node_modules/.bin/webpack \
--config /tmp/nix-build-gitlab-assets-13.0.9.drv-2/source/config/webpack.config.js --bail
Update 2:
- On source VPS
/proc/cpuinfo
shows 32 cores on the VPS - On target VPS,
/proc/cpuinfo
shows only 8 cores (as expected). The target VPS is also a nixos on node15.prg. - On target VPS in container (created via nixops),
/proc/cpuinfo
shows 32 cores.
I don't know what's the difference between source VPS and target VPS, I would expect the same result on both.
Well, 32 cores on the sources VPS (called builder) was a bug in vpsAdmin which somehow let you disable CPU limit altogether... I've set it again, so you can give it another go...
But the issue is that nproc
will report number of CPUs from the host and not the VPS, so wrong /proc/cpuinfo
might not have caused this, it depends which way to check available CPUs the application uses.
Oh and when you're using nixops, the target VPS doesn't matter at all. Everything is built on the source and then only copied to the target, so you might ignore the target when you have problems with building.
My account was allowed to bypass the limits for some unknown reason. It was resolved in ticket #34044.
I would expect I can overwrite nproc
with --cores
or NIX_BUILD_CORES
, so it may not be a problem if that one is not correct. webpack
shouldn't care, I will give it a try. For number of cpus webpack
uses something like:
var _os = require("os");
console.log('os', _os)
console.log('cpus', _os.cpus());
strace
shows that it reads /proc/cpuinfo
.
It's not related to #34044, this was a new one vpsfreecz/vpsadmin@53ad85a
Yep, it always depends on the build system if it uses autodetection or if it can be configured... on NixOS, you can set options nix.maxJobs
and nix.buildCores
on the source VPS to have it persistent.
I confirm webpack
now spawns only 4 node
s instead of 32. Closing. Thanks for the fix and hints.
The webpack
compilation is still killed by OOM, but now it looks like a greedy webpack
.