Build fails with `sh: 1: cargo: not found`
kariae opened this issue · 62 comments
Maintainer's Note: This has become a meta-issue of multiple bugs all causing the host toolchain to not be mounted in the container, and includes:
- Docker in Docker (#387)
- Nix Store (#524)
- Remote Docker support (#273, #785)
- WSL2
- WSL2 Docker-in-Docker (#728)
- Running cross in parallel (#496)
- Docker-in-Docker Fails to mount host dirs (#1283)
The build fails on a Docker container with sh: 1: cargo: not found, the Docker container is run with a custom image that contains Docker with a wrapper to have DinD working, and Rust nightly docker run -it -v pwd:/var/tmp -v /var/run/docker.sock:/var/run/docker.sock kariae/docker-rust-nightly sh the command fails with sh: 1: cargo: not found
I am also encountering this issue.
Nothing special about my setup, though.
[evanjs@nixentoo:~/src/scratch/sdl2test]$ cross build --target x86_64-pc-windows-gnu
Unable to find image 'japaric/x86_64-pc-windows-gnu:v0.1.14' locally
v0.1.14: Pulling from japaric/x86_64-pc-windows-gnu
b8a43e1b7983: Pull complete
5c5185a06822: Pull complete
8458c4a4e317: Pull complete
6d0388b275d4: Pull complete
56603a33e04b: Pull complete
2cedd67f6904: Pull complete
5d40de413ec5: Pull complete
fbd3dda3b3f5: Pull complete
ee23f3a81425: Pull complete
22d70475d56a: Pull complete
cc2c4c043bf3: Pull complete
ace8770c3426: Pull complete
Digest: sha256:b164f497e18f89133de814898d838603b07a6b1feb80b06271ddb3fb86128cc8
Status: Downloaded newer image for japaric/x86_64-pc-windows-gnu:v0.1.14
sh: 1: cargo: not found
Using NixOS 19.03 on 4.19.24
Docker: 18.09.2
Cross: 0.1.14
cargo 1.34.0-nightly (5c6aa46e6 2019-02-22)
I checked the image and I didn't find the installation of cargo which is a little bit tricky, since we use it to build the binaries ?!
I am encountering the same problem. I have just tried running cross build --target aarch64-unknown-linux-musl and got the following output
Unable to find image 'japaric/x86_64-unknown-linux-musl:v0.1.14' locally
v0.1.14: Pulling from japaric/x86_64-unknown-linux-musl
8d51e945c435: Pull complete
d8d21cdaac84: Pull complete
99b5ff394311: Pull complete
3ab3190bc52d: Pull complete
8f2bd7981235: Pull complete
ba0de5bbf3ef: Pull complete
b8380c29d163: Pull complete
e75a875c5cd4: Pull complete
e10da4c5e540: Pull complete
26440e989b5e: Pull complete
b87e83e92416: Pull complete
2e920f245507: Pull complete
Digest: sha256:524b2d33f250ce4e243397b26aab5be7e90d5413b047cc1b9461a62b42b603e6
Status: Downloaded newer image for japaric/x86_64-unknown-linux-musl:v0.1.14
sh: 1: cargo: not found
like @evanjs I am also running on NixOS (18.03.133402.cb0e20d6db9 (Impala))
I'm also hit by
sh: 1: cargo: not found
failed to resize tty, using default size
sh: 1: cargo: not found
On Linux, NixOS 19.03
Same problem here. It appears to be limited to executing cross within a docker image. I'm curious if anyone has workarounds?
The problem is that the image uses the host's own Rust install, which in some cases (e.g. NixOS) cannot be run on Ubuntu. You can work around it by building an image with its own Rust install, for example:
FROM japaric/x86_64-unknown-freebsd:v0.1.4
RUN apt-get update && apt-get install -y wget
RUN mkdir -m777 /opt/rust /opt/cargo
ENV RUSTUP_HOME=/opt/rust CARGO_HOME=/opt/cargo PATH=/opt/cargo/bin:$PATH
RUN wget --https-only --secure-protocol=TLSv1_2 -O- https://sh.rustup.rs | sh /dev/stdin -y
RUN rustup target add x86_64-unknown-freebsd
RUN printf '#!/bin/sh\nexport CARGO_HOME=/opt/cargo\nexec /bin/sh "$@"\n' >/usr/local/bin/sh
RUN chmod +x /usr/local/bin/shRust is installed outside ~ so the user the build runs as can access it. About the last two lines, I wasn't able to override CARGO_HOME from the shell profile so I overrode the shell instead.
Not pretty, but gets the job done.
Using something from the host breakes many of the otherwise strong guarantees alike "runs fine with me, will run fine with you" docker has to offer. IMO this should be at least considered to be changed.
Have you tried installing cross from master?
I had still to make some workaround for using cross in gitlab-ci even when using the master branch.
But I managed to do it, you can find information in this comment: #273 (comment)
This probably does not apply for nixOS users. But for the OP surely :)
Same issue was found on circle-ci build: https://circleci.com/gh/shadowsocks/shadowsocks-rust/5
Could that be fixed by your script, too? @AluisioASG
# cargo install --git https://github.com/rust-embedded/cross.git
Updating git repository `https://github.com/rust-embedded/cross.git`
Installing cross v0.2.0-alpha.1 (https://github.com/rust-embedded/cross.git#2f284acd)
<removed>
Replaced package `cross v0.1.16` with `cross v0.2.0-alpha.1 (https://github.com/rust-embedded/cross.git#2f284acd)` (executable `cross`)
# cross build --target x86_64-pc-windows-gnu
Unable to find image 'rustembedded/cross:x86_64-pc-windows-gnu' locally
x86_64-pc-windows-gnu: Pulling from rustembedded/cross
<removed>
Digest: sha256:cdc542fe0a4b5cf89817388bcb4769f430a2979281add6f5e8fb26baf9f94a9a
Status: Downloaded newer image for rustembedded/cross:x86_64-pc-windows-gnu
sh: 1: cargo: not found
I tried to install cargo from master but running into the same issue. I'm using WSL. Any suggestions on how to troubleshoot further?
Did run into a similar issue with development containers and Docker in Docker setup. I created a small POC (PR #387) that tries to find the corrects paths on the host machine when spinning up the child container.
I can confirm this issue, it also happens on armv7-unknown-linux-musleabihf-0.2.0
shouldn't rust be installed in those container images ? or I misunderstood the way this tool works
FROM rustembedded/cross:armv7-unknown-linux-musleabihf-0.2.0
ENV PATH="/root/.cargo/bin:$PATH"
RUN curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain stable
same problem, for some reason it worked a while ago. but not now (macOS)
I can confirm as well, started happening a couple of days ago. Here I'm not even using an official rustembedded docker image, but a relatively simple Alpine linux one (see rust-bio/rust-htslib#193 if you need more details to reproduce and/or context):
$ rustup show
Default host: x86_64-unknown-linux-gnu
rustup home: /home/limsadmin/.rustup
installed targets for active toolchain
--------------------------------------
x86_64-unknown-linux-gnu
x86_64-unknown-linux-musl
active toolchain
----------------
stable-x86_64-unknown-linux-gnu (default)
rustc 1.42.0 (b8cedc004 2020-03-09)
$ cat Cross.toml
[target.x86_64-unknown-linux-musl]
image = "brainstorm/rust-htslib-musl-alpine:latest"
$ cat docker/Dockerfile.alpine
FROM alpine:edge
RUN apk add --update wget git g++ build-base automake autoconf openssl openssl-dev curl-dev bzip2-dev xz-dev zlib-dev rustup && \
rm -rf /var/cache/apk/*
WORKDIR /root
RUN rustup-init -y
ENV PATH "/root/.cargo/bin:$PATH"
RUN rustup target add x86_64-unknown-linux-musl
$ cross build --target x86_64-unknown-linux-musl
sh: cargo: not found@reitermarkus Tagging you, as there seems to be official status on this, I hope you can provide more info.
Why is this issue still unfixed? @AluisioASG solution is a workaround that works, but it is rather inconvenient. How does cross pickup the host systems cargo? And why doesn't this mechanism work for some people (like me)?
Would it be possible to have a cargo with the matching target already added in the containers as a fallback? So that if possible host cargo is used, if not container image's cargo?
I am hitting this issue as well, running on MacOS 10.15 and docker-in-dockering it. If I change cross to the master branch, it fails out with "Could not find directory of OpenSSL installation," but openssl-dev is installed.
I have the same problem running on rust:1.44-alpine docker image.
After running apk add cmake make gcc libtool musl-dev g++
I see /bin/sh: 3: cargo: not found
This works perfectly fine on macOS.
I also ran into the problem on nixOS while trying to cross compile for x86_64-unknown-linux-musl.
I ended up using https://github.com/emk/rust-musl-builder#what-is-this since it worked out of the box for me.
For those running cross from within a Docker image, see https://github.com/rust-embedded/cross#docker-in-docker.
An example of doing this interactively using the official rust image:
docker-host$ docker run -it -e CROSS_DOCKER_IN_DOCKER=true -v /var/run/docker.sock:/var/run/docker.sock -v $(pwd):/usr/src/myproj -w /usr/src/myproj --rm rust bash
root@c7fddc07d311:/usr/src/myproj# wget https://download.docker.com/linux/static/stable/x86_64/docker-19.03.12.tgz
root@c7fddc07d311:/usr/src/myproj# tar -C /usr/local/bin --strip-components=1 -xf docker-19.03.12.tgz docker/docker
root@c7fddc07d311:/usr/src/myproj# cargo install cross
root@c7fddc07d311:/usr/src/myproj# cross build --target aarch64-unknown-linux-gnu
While working on this pull request (#467), I started encountering this message. I had just recently changed the path configuration for fish where I was placing /usr/local/bin before the one for cargo at ~/.cargo/bin/. Removing that entry seemed to clear up the issue. (As a side note, I'm on OSX running 10.14.6)
P.S. I'll also add I had two versions of Rust installed. One via brew and one via rustup. I uninstalled the one installed with Homebrew. That was likely the issue all along.
I am having trouble to get cross to work with circleci.
I have following config.yml:
version: 2.1
executors:
ci-runner:
docker:
- image: cimg/rust:1.47.0
jobs:
build:
docker:
- image: cimg/rust:1.47.0
environment:
CROSS_DOCKER_IN_DOCKER: true
steps:
- checkout
- restore_cache:
key: v1-cargo-cross
- setup_remote_docker:
version: 19.03.13
docker_layer_caching: true
- run:
name: Install protobuf dependencies
command: |
sudo apt-get update
sudo apt install protobuf-compiler
- run:
name: Install cargo cross
command: |
set -x
CROSS=/home/circleci/.cargo/bin/cross
if [ -f "$CROSS" ]; then
echo "cargo cross already installed"
else
cargo install cross
fi
- save_cache:
key: v1-cargo-cross
paths: /home/circleci/.cargo/bin/cross
# - run:
# name: Build for x86_64
# command: cargo build
- run:
name: Build for aarch64
command: docker run -e CROSS_DOCKER_IN_DOCKER=true -v /var/run/docker.sock:/var/run/docker.sock -v $(pwd):/home/circleci/project -w /home/circleci/project rustembedded/cross:aarch64-unknown-linux-musl cross build --target aarch64-unknown-linux-musl
workflows:
version: 2
app:
jobs:
- build
And I get error that cross is not found:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"cross\": executable file not found in $PATH": unknown.
Maybe there is some obvious error in this docker command?
@andresv rustembedded/cross:aarch64-unknown-linux-musl doesn't contain a cross executable, only the corresponding cross toolchain. You already installed cross in your cimg/rust:1.47.0 container, so just replace the whole docker run... command with cross build --target aarch64-unknown-linux-musl.
defining variable CROSS_DOCKER_IN_DOCKER=true inside of container helped me
thanks to @jjlin
It appears to be added in this commit f2c70b3 (present in v0.2.1)
Does it mean this ticket could be closed ?
@ndusart No. I've had this issue without running docker in docker, so fixing docker in docker does not solve the entirety of this issue.
I seem to be getting this error when building for multiple targets at the same time
I'm getting this same issue on NixOS 20.09:
$ cross build --target x86_64-unknown-linux-musl
Unable to find image 'rustembedded/cross:x86_64-unknown-linux-musl-0.2.1' locally
x86_64-unknown-linux-musl-0.2.1: Pulling from rustembedded/cross
d7c3167c320d: Pull complete
131f805ec7fd: Pull complete
322ed380e680: Pull complete
6ac240b13098: Pull complete
64a715cd0a53: Pull complete
6dd11a4cf760: Pull complete
1914c0932fa8: Pull complete
0f453323eca7: Pull complete
d8b3882bf295: Pull complete
1f4170793a2e: Pull complete
b01bc4f3cef7: Pull complete
f4e0966a43d2: Pull complete
Digest: sha256:e906258406a86a350339146feb8f33614d880151481eb303c9270d9db26aaa60
Status: Downloaded newer image for rustembedded/cross:x86_64-unknown-linux-musl-0.2.1
sh: 1: cargo: not found
As am i, also NixOS 20.09:
Unable to find image 'rustembedded/cross:x86_64-pc-windows-gnu-0.2.1' locally
x86_64-pc-windows-gnu-0.2.1: Pulling from rustembedded/cross
d7c3167c320d: Pull complete
131f805ec7fd: Pull complete
322ed380e680: Pull complete
6ac240b13098: Pull complete
d7bf0568dabc: Pull complete
981aecd3bf48: Pull complete
6697c741376d: Pull complete
cc1566a6c043: Pull complete
7e4da0436841: Pull complete
6a8d255df934: Pull complete
8928e758bdd0: Pull complete
f5186e3ba9b6: Pull complete
8f352a8fa374: Pull complete
4ad1b6ba757d: Pull complete
Digest: sha256:5b195dffc3876e155bb95aac347be1440d1cd5bc85fb46c7d9b3feeed2b16d8c
Status: Downloaded newer image for rustembedded/cross:x86_64-pc-windows-gnu-0.2.1
sh: 1: cargo: not foundHit this as well. macOS 11.2.3 on amd64
% cross build --target x86_64-unknown-linux-musl
info: syncing channel updates for '1.50.0-x86_64-apple-darwin'
1.50.0-x86_64-apple-darwin unchanged - rustc 1.50.0 (cb75ad5db 2021-02-10)
info: checking for self-updates
Unable to find image 'rustembedded/cross:x86_64-unknown-linux-musl-0.2.1' locally
x86_64-unknown-linux-musl-0.2.1: Pulling from rustembedded/cross
d7c3167c320d: Pull complete
131f805ec7fd: Pull complete
322ed380e680: Pull complete
6ac240b13098: Pull complete
64a715cd0a53: Pull complete
6dd11a4cf760: Pull complete
1914c0932fa8: Pull complete
0f453323eca7: Pull complete
d8b3882bf295: Pull complete
1f4170793a2e: Pull complete
b01bc4f3cef7: Pull complete
f4e0966a43d2: Pull complete
Digest: sha256:e906258406a86a350339146feb8f33614d880151481eb303c9270d9db26aaa60
Status: Downloaded newer image for rustembedded/cross:x86_64-unknown-linux-musl-0.2.1
sh: 1: cargo: not foundI'm getting this issue as well. What's the point of the Docker image if the tool isn't preinstalled? Is this project dead?
I hit the same issue but with a different root cause. I just want to share some notes that might help to people seeing this problem.
- The docker images used by
crossdoesn't include rust/cargo or other tool. These image provides the libraries required for cross-compiling. - The
crosstool installs the requested toolchain in the system and then when docker is run some volumes are mounted to make rust visible inside the container. - The
cargo: not founderror is because some of these volume mounts is not working correctly.
Adding the --verbose flag to the cross build command helps to get the docker command used to launch the container. Try to run the same command from your terminal and start checking which of the volumes is failing.
+1 on current NixOS unstable 21.11pre304626.8ecc61c91a5 (Porcupine)
I'm also running into this on NixOS, current stable 21.05.
same issue here. NixOS v21.11
when installing rust, I use bash instead of sh
and ENV PATH="/root/.cargo/bin:${PATH}" is necessary.
this works fine for me
This issue is fixed on main for nixos. There are a lot of different issues reported here, but many seem to be related to Nix.
If you are experiencing the error message, please first try installing cross from source, and then if that fails with the same error message, add -vv and paste the entire debug output inside <details><summary>Details</summary> ... </details>
@ppdogg I'm not sure what you mean and how you solved the issue, did you add that to the dockerfile? If you did, that's an error, as you shouldn't be using the cargo in the dockerfile, but your rustup toolchain install x86_64-unknown-linux-gnu image is what should be mounted to /cargo
Sidenote: That it's still available is interesting though, we should probably remove that (unless it's needed for xargo maybe?)
@Emilgardis thanks for replying. I am not so sure that we face the same situation.
here is the solution I referred: https://stackoverflow.com/questions/49676490/when-installing-rust-toolchain-in-docker-bash-source-command-doesnt-work
I was building a docker image in a modified centos. the script says:
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --default-toolchain nightly-2021-12-23 -y
ENV PATH "/root/.cargo/bin:${PATH}"
but an error comes:
/bin/sh: 1: cargo: not found
I soon realized that rust may not be successfully installed, and the installation script in rust official site doesn't work either, which is:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
In the end, I changed the script. using bash and adding =:
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | bash -s -- --default-toolchain nightly-2021-12-23 -y
ENV PATH="/root/.cargo/bin:${PATH}"
@ppdogg I fail to see how that is related to cross
Hi I am on nix 22.05. I have the same issue. Here is the output of cross build --target armv7-unknown-linux-gnueabihf -vv
+ "rustc" "--print" "sysroot"
+ "rustup" "toolchain" "list"
+ "rustup" "target" "list" "--toolchain" "nightly-x86_64-unknown-linux-gnu"
+ "rustup" "component" "list" "--toolchain" "nightly-x86_64-unknown-linux-gnu"
+ "/run/current-system/sw/bin/docker" "run" "--userns" "host" "-e" "PKG_CONFIG_ALLOW_CROSS=1" "--rm" "--user" "1000:100" "-e" "XARGO_HOME=/xargo" "-e" "CARGO_HOME=/cargo" "-e" "CARGO_TARGET_DIR=/target" "-e" "USER=david" "-e" "CROSS_RUNNER=" "-v" "/home/david/.xargo:/xargo:Z" "-v" "/home/david/.cargo:/cargo:Z" "-v" "/cargo/bin" "-v" "/home/david/Documents/Projects/Rust/cross-test:/project:Z" "-v" "/home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu:/rust:Z,ro" "-v" "/home/david/Documents/Projects/Rust/cross-test/target:/target:Z" "-w" "/project" "-i" "-t" "ghcr.io/cross-rs/armv7-unknown-linux-gnueabihf:main" "sh" "-c" "PATH=$PATH:/rust/bin cargo build --target armv7-unknown-linux-gnueabihf -vv"
sh: 1: cargo: not found
Forgot to mention cross was installed from git (latest commit)
this solution worked
FROM ghcr.io/cross-rs/arm-unknown-linux-gnueabihf:main
RUN apt-get update && apt-get install -y wget
RUN mkdir -m777 /opt/rust /opt/cargo
ENV RUSTUP_HOME=/opt/rust CARGO_HOME=/opt/cargo PATH=/opt/cargo/bin:$PATH
RUN wget --https-only --secure-protocol=TLSv1_2 -O- https://sh.rustup.rs | sh /dev/stdin -y
RUN rustup target add arm-unknown-linux-gnueabihf
RUN printf '#!/bin/sh\nexport CARGO_HOME=/opt/cargo\nexec /bin/sh "$@"\n' >/usr/local/bin/sh
RUN chmod +x /usr/local/bin/sh@IllustratedMan-code good that it worked for you, but that solution is not how cross is supposed to work (as I alluded to here and mentioned here), cross is supposed to mount your system sysroot for x86_64-unknown-linux-gnu into /
For nix specifically, the issue should have been fixed with #524
We have to continue and look into this, but its hard when we're not able to reproduce the issue
I have the same issue in WSL2 (Ubuntu 20.04) running in Windows 11. Running Docker via Rancher Desktop in Windows and confirmed DinD works with docker run --rm -i hello-world.
Reproducing it like this:
Start build container that includes Docker
docker run --rm -it --entrypoint bash \
-e CROSS_DOCKER_IN_DOCKER=true \
-v /var/run/docker.sock:/var/run/docker.sock \
gcr.io/cloud-builders/dockerInstall Rust and cross, create project
apt update && apt install build-essential -y
curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain stable
source $HOME/.cargo/env
cargo install cross
cargo init hello-world && cd hello-worldTry to build with cross
cross build
info: downloading component 'rust-src'
info: installing component 'rust-src'
sh: 1: cargo: not foundEDIT: The exact same steps works on Windows 11 and Manjaro 21.2.6 with Linux 5.17.
I checked the image and I didn't find the installation of
cargowhich is a little bit tricky, since we use it to build the binaries ?!
The docker images use the host cargo, so it is not installed on the image, and the directory is mounted to /cargo.
@Emilgardis I just took a look at #524 . It seems to rely on a NIX_STORE environment variable to determine the location of the nix store. The NIX_STORE variable is not set by nix normally as far as I can tell. The default location is /nix/store so I'll try setting the NIX_STORE variable and see if it fixes it.
Aha, that was not my understanding
Running cross with the NIX_STORE environment variable set to /nix/store seems to work for me on NixOS.
Some more context to the WSL issue without really having a solution.
I had a look at the code for when cross starts a docker container and if you use DOCKER_IN_DOCKER=true it inspects the current container (docker inspect $HOSTNAME) to determine the volumes and tries to mount them, something like /var/lib/docker/overlay2/xxx/merged/root/.cargo:/cargo:Z. This is needed since afaik you won't be able to mount volumes from Docker to Docker but instead have to reference the host machine/volume (ref).
However, when running Rancher Desktop (and I think Docker Desktop as well?) in combination with WSL, it sets up two different distros to host Docker:
$ wsl --list -v
NAME STATE VERSION
* Ubuntu Running 2
rancher-desktop Running 2
rancher-desktop-data Running 2The container data volume does not exist in Ubuntu or rancher-desktop, however it exists in rancher-desktop-data. This means that if I look at the output for docker inspect $HOSTNAME and .[0].GraphDriver.Data, it will refer to paths which does not exist and is not accessible in the current container that tries to run cross.
I usually don't use Windows or WSL so I need to read up more on how volume mounts + Windows + WSL + Docker in Docker works but right now it looks as simple as that the cross container that gets launched from the Docker container running in WSL basically can't mount any toolchain successfully which also explains why cargo is not found.
This feels like a very WSL specific issue so let me know if you want me to open a new issue to track this.
This feels like a very WSL specific issue so let me know if you want me to open a new issue to track this.
Yes please do!
I can confirm this issue, it also happens on
armv7-unknown-linux-musleabihf-0.2.0shouldn't rust be installed in those container images ? or I misunderstood the way this tool works
FROM rustembedded/cross:armv7-unknown-linux-musleabihf-0.2.0 ENV PATH="/root/.cargo/bin:$PATH" RUN curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain stable
No, although we will be shipping an image with cross pre-installed in #878.
I believe the only issue left is running cross in Rancher Desktop and WSL2. We've fixed:
NIX_STORE- Docker in docker
- Remote docker support
- Regular WSL2 issues
I am having trouble to get cross to work with circleci. I have following
config.yml:And I get error that
crossis not found:docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"cross\": executable file not found in $PATH": unknown.Maybe there is some obvious error in this docker command?
Yeah, the images do not come with cross-installed, you are supposed to use the host tooolchain and you are trying to use cross inside the image. You are supposed to run cross on the host which mounts the Rust toolchain inside the image and comes pre-installed with the necessary linkers/C compilers for cross-compilation. Just change the command to CROSS_DOCKER_IN_DOCKER=true build --target aarch64-unknown-linux-musl and it should work. Make sure the image itself was mounted with -v /var/run/docker.sock:/var/run/docker.sock or a correct mount from the host Docker socket to the image Docker socket.
The actual mount point seems to be /run/desktop/mnt/host/$drive/$path, however, this doesn't seem very fruitful. A better approach seems to be /mnt/wsl/docker-desktop-bind-mounts/Ubuntu-22.04/9d1f950fb5631e36b799853d4e1b9f9150a092eba2d6fce54dc60f78297cd082/hell, but um, this doesn't seem super reliable for reasons. This isn't the current hostname either, nor the current ID from the storage driver. So, I don't think this is really feasible, since it requires the current WSL2 machine name, and some seemingly random identifier.
The remaining issues for cargo not being found have their own independent issues, and there are better used to track the status of the container engine failing to mount host volumes.
Just created a pull request that should fix the NIX_STORE problem.
I'm seeing these issue on mac.
@bnheise please open a new issue for your issue. For us to best help you/identify the problem, include the output of the cross command with -v added
Hi @Emilgardis Thanks for the direction. The issue is no longer occurring for me. I'm not sure why yet though.
I have an idea why this is happening on Docker in Docker runs.
If /var/lib/docker is not bind mounted to the outer Docker instance, when cross goes to forward some of those mounts, there is nothing to forward to the inner Docker instance:
rustuser@rust-build:~/workspace$ cross build -vv --target aarch64-unknown-linux-gnu
+ cargo metadata --format-version 1 --filter-platform aarch64-unknown-linux-gnu
+ rustc --print sysroot
+ rustup toolchain list
+ rustup target list --toolchain 1.70.0-x86_64-unknown-linux-gnu
+ rustup component list --toolchain 1.70.0-x86_64-unknown-linux-gnu
+ /usr/bin/docker
+ /usr/bin/docker run --userns host -e 'PKG_CONFIG_ALLOW_CROSS=1' -e 'XARGO_HOME=/xargo' -e 'CARGO_HOME=/cargo' -e 'CARGO_TARGET_DIR=/target' -e 'CROSS_RUNNER=' -e TERM -e 'USER=rustuser' -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace/builddir/library:/var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace/builddir/library --rm --user 1000:1000 -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/.xargo:/xargo:z -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/opt/cargo:/cargo:z -v /cargo/bin -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace:/var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace:z -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/usr/local/rustup/toolchains/1.70.0-x86_64-unknown-linux-gnu:/rust:z,ro -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace/target:/target:z -w /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace -i -t ghcr.io/cross-rs/aarch64-unknown-linux-gnu:0.2.5 sh -c 'PATH=$PATH:/rust/bin cargo build -vv --target aarch64-unknown-linux-gnu'
sh: 1: cargo: not found
+ rustup component list --toolchain 1.70.0-x86_64-unknown-linux-gnu
If you run the container above and drop into a shell, you can examine the paths available in /var/lib and see what is missing.
rustuser@rust-build:~/workspace$ /usr/bin/docker run --userns host -e 'PKG_CONFIG_ALLOW_CROSS=1' -e 'XARGO_HOME=/xargo' -e 'CARGO_HOME=/cargo' -e 'CARGO_TARGET_DIR=/target' -e 'CROSS_RUNNER=' -e TERM -e 'USER=rustuser' -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace/builddir/library:/var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace/builddir/library --rm --user 1000:1000 -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/.xargo:/xargo:z -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/opt/cargo:/cargo:z -v /cargo/bin -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace:/var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace:z -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/usr/local/rustup/toolchains/1.70.0-x86_64-unknown-linux-gnu:/rust:z,ro -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace/target:/target:z -w /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace -i -t ghcr.io/cross-rs/aarch64-unknown-linux-gnu:0.2.5 sh
$ ls
target
$ id
uid=1000 gid=1000 groups=1000
$ ls /
bin boot cargo common.sh dev etc home lib lib.sh lib64 linux-image.sh linux-runner media mnt opt proc qemu root run rust sbin srv sys target tmp usr var xargo
rustuser@rust-build:~/workspace$ ls -l /var/lib
total 48
drwxr-xr-x 1 root root 4096 Jun 28 14:25 apt
drwxr-xr-x 1 root root 4096 Jun 28 14:26 dpkg
drwxr-xr-x 2 root root 4096 Feb 22 10:51 git
drwxr-xr-x 2 root root 4096 Apr 2 13:55 misc
drwxr-xr-x 2 root root 4096 Jun 12 02:00 pam
drwxr-xr-x 2 root root 4096 Jun 13 05:29 python
drwxr-xr-x 3 root root 4096 Jun 28 14:25 sudo
drwxr-xr-x 1 root root 4096 Jun 13 05:29 systemd
drwxr-xr-x 3 root root 4096 Jun 13 05:29 ucf
rustuser@rust-build:~/workspace$ ls -l /var/lib/docker
ls: cannot access '/var/lib/docker': No such file or directory
I don't have a solution to this yet. It should be possible to bind mount /var/lib/docker, but the root:root permissions will be an issue. It doesn't really make sense to me why cross reuses the host system Rust tools.
cross reuses the hosts rust tools so that you can specify what rust version to run, and so that we don't have to follow the 6 week releases of rust.
For docker in docker, are you using the correct env vars as mentioned in the readme (CROSS_CONTAINER_IN_CONTAINER=true)? If you are and it still doesn't work, thats a bug unrelated to this issue, feel free to open a new issue or reach out on our matrix chat (also found in our readme :D )
Hmm. I think it creates a dependency that might be more easily folded into a single container or deriving from rust:latest. For people running CI servers, it means having to install Rust on the server directly or fiddling with this Docker in Docker situation.
I purposefully don't install Rust in the outermost host system I use for development. And I'm not sure how many corporate users are aiming for anything newer than the stable release anyways, it's usually the opposite 😼.
I am indeed using CROSS_CONTAINER_IN_CONTAINER=true. Let me go ahead and put the details in a new issue and link it here -- #1283
If you experience this, please see the other issues mentioned in the first comment, open a new issue or reach out on our matrix chat!
Thank you!