rust-lang/cargo

Cargo eating all RAM during aarch64 build under Docker on x86-64

nazar-pc opened this issue · 9 comments

nazar-pc commented

Problem

I'm trying to build an application, this time to build aarch64 container image on x86-64 machine.

My machine is a beefy 5900X with 128G of RAM, but my system runs out of memory (Cargo eats it all) when cross-compiling aarch64 container.

Steps

  1. Clone this branch: https://github.com/nazar-pc/subspace/tree/cargo-memory-bug-aarch64
  2. Have Docker installed on Linux x86-64 machine
  3. Run docker buildx build --platform linux/arm64 -t test -f Dockerfile-farmer .
  4. Observe system freezing/crashing because of running out of memory

Possible Solution(s)

No response

Notes

x86-64 build on the same machine works fine with ~20G of system memory.

Not entirely sure if this is Cargo's fault or QEMU or something else, but I used similar QEMU-based setups before and never seen anything remotely like this.

Version

cargo 1.60.0-nightly (c082648 2022-02-08)
release: 1.60.0-nightly
commit-hash: c082648646cbb2be266df9ecbcdc253058158d68
commit-date: 2022-02-08
host: aarch64-unknown-linux-gnu
libgit2: 1.3.0 (sys:0.13.23 vendored)
libcurl: 7.80.0-DEV (sys:0.4.51+curl-7.80.0 vendored ssl:OpenSSL/1.1.1m)
os: Linux [64-bit]

UPD: Tried the most recent version, still the same issue:

cargo 1.62.0-nightly (dba5baf 2022-04-13)
release: 1.62.0-nightly
commit-hash: dba5baf4345858c591517b24801902a062c399f8
commit-date: 2022-04-13
host: aarch64-unknown-linux-gnu
libgit2: 1.4.2 (sys:0.14.2 vendored)
libcurl: 7.80.0-DEV (sys:0.4.51+curl-7.80.0 vendored ssl:OpenSSL/1.1.1m)
os: Linux [64-bit]
epage commented

#8405 is one potential solution for this.

nazar-pc commented

Hm, I feel like this is something different because I only observe this issue in above scenario with QEMU-based emulation. Let me know if I can add some instrumentation to help debugging this (though reproduction steps should be reliable too as everything is containerized).

nazar-pc commented

I'm fairly certain that memory consumption happens during repo cloning (dependencies from git), before doing any compilation

weihanglo commented

If it is related to git clone, perhaps set the config net.git-fetch-with-cli = true to see whether libgit2 is the suspect.

  • 👍5
  • ❤️7
nazar-pc commented

Thank you @weihanglo, it helped!

  • 👍3
Flawm commented

I'm seeing the exact same behavior as OP, however setting the net git fetch flag didn't seem to reduce the memory consumption

In my case, I was able to get around it by binding my cargo caches into the container:

    -v $HOME/.cargo/git:/usr/local/cargo/git \
    -v $HOME/.cargo/registry:/usr/local/cargo/registry \

I suppose the docker-compose.yml equivalent would work as well, but in some CI scenarios this may not be possible

  • ❤️1
antonputra commented

Resolved by creating config file /projects/foo/.cargo/config.toml. (Tried to compile amd64 from mac m1)

[net]
git-fetch-with-cli = true
  • ❤️5
msrd0 commented

I just ran into this issue as well trying to execute cargo fetch on an alpine aarch64 guest system running on an x86_64 archlinux host. It never made it past cloning the crates.io index.