Unable to use cross build --offline in docker-in-docker environment

[fjnuLuis]

Checklist

• I've looked through the issues and pull requests for similar reports

Describe your issue

How To Step On This Bug

• In a private project, I tried to use cross build in docker, and called cargo fetch in advance to download the cache. After that, every time I compile, I will open --offline mode and do not connect to the Internet.
• I have made sure that all the following environment variables have been set to ensure that the previously downloaded cache is copied into the new docker container：
  • CROSS_REMOTE_COPY_REGISTRY=1
  • CROSS_REMOTE=1
  • CROSS_REMOTE_COPY_CACHE=1
• Even after completing the above work, the previously downloaded cache cannot be found in the new docker container. The following is the log printed by turning on --verbose during compilation：

$ CROSS_REMOTE_COPY_REGISTRY=1 CROSS_REMOTE=1 CROSS_REMOTE_COPY_CACHE=1 cross build --offline --target aarch64-unknown-linux-gnu --verbose
+ cargo metadata --format-version 1 --filter-platform aarch64-unknown-linux-gnu
+ rustc --print sysroot
+ rustup toolchain list
+ rustup target list --toolchain stable-x86_64-unknown-linux-gnu
+ rustup component list --toolchain stable-x86_64-unknown-linux-gnu
+ /usr/bin/docker
+ /usr/bin/docker volume inspect cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8
+ /usr/bin/docker ps -a --filter 'name=cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44' --format {{.State}}
+ /usr/bin/docker run --userns host --name cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44 --rm -v /cross -v /cross/cargo/bin -d -t ghcr.io/cross-rs/aarch64-unknown-linux-gnu:0.2.5
39638b353eb878d5855061f3b5bab4ba1127b16473caaec1adbb3a6cf15a0477
+ /usr/bin/docker cp -a /root/.cargo cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/cargo
Successfully copied 70.3MB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/cargo
+ /usr/bin/docker exec cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44 sh -c 'mkdir -p '\''/cross/rust/lib/rustlib'\'''
+ /usr/bin/docker cp -a /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/rust
Successfully copied 49.1MB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/rust
+ /usr/bin/docker cp -a /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/libexec cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/rust
Successfully copied 1.25MB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/rust
+ /usr/bin/docker cp -a /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/etc cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/rust
Successfully copied 13.8kB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/rust
+ /usr/bin/docker cp -a /root/.local/share/cross-rs/tmp/.tmpgjMtjJ/lib cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/rust
Successfully copied 299MB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/rust
+ /usr/bin/docker cp -a /root/.local/share/cross-rs/tmp/.tmpBRqumm/lib cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/rust
Successfully copied 855kB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/rust
+ /usr/bin/docker cp -a /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/rust/lib/rustlib
Successfully copied 149MB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/rust/lib/rustlib
+ /usr/bin/docker cp -a /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/rust/lib/rustlib
Successfully copied 253MB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/rust/lib/rustlib
+ /usr/bin/docker cp -a /workspace cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/project
Successfully copied 9.22kB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/project
+ /usr/bin/docker exec cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44 sh -c 'set -e pipefail
set -x
chown -R 0:0 /cross
prefix="/cross"

symlink_recurse() {
    for f in "${1}"/*; do
        dst=${f#"$prefix"}
        if [ -f "${dst}" ]; then
            echo "invalid: got unexpected file at ${dst}" 1>&2
            exit 1
        elif [ -d "${dst}" ]; then
            symlink_recurse "${f}"
        else
            ln -s "${f}" "${dst}"
        fi
    done
}

symlink_recurse "${prefix}"
'
+ chown -R 0:0 /cross
+ prefix=/cross
+ symlink_recurse /cross
+ dst=/cargo
+ [ -f /cargo ]
+ [ -d /cargo ]
+ ln -s /cross/cargo /cargo
+ dst=/project
+ [ -f /project ]
+ [ -d /project ]
+ ln -s /cross/project /project
+ dst=/rust
+ [ -f /rust ]
+ [ -d /rust ]
+ ln -s /cross/rust /rust
+ /usr/bin/docker exec --user 0:0 -e 'PKG_CONFIG_ALLOW_CROSS=1' -e 'XARGO_HOME=/xargo' -e 'CARGO_HOME=/cargo' -e 'CARGO_TARGET_DIR=/target' -e 'CROSS_RUNNER=' -e TERM -e CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_RUNNER -e http_proxy -e CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER -e https_proxy -e 'USER=root' -w /project cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44 sh -c 'PATH=$PATH:/rust/bin cargo build --offline --target aarch64-unknown-linux-gnu --verbose --target-dir /cross/project/target'
error: no matching package named `log` found
location searched: registry `crates-io`
required by package `playground v0.1.0 (/cross/project)`
As a reminder, you're using offline mode (--offline) which can sometimes cause surprising resolution failures, if this error is too confusing you may wish to retry without the offline flag.
+ /usr/bin/docker exec cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44 bash -c '[[ -d '\''/cross/project/target'\'' ]]'
+ /usr/bin/docker cp -a cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/project/target /workspace
Successfully copied 5.12kB to /workspace
+ /usr/bin/docker stop cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44 --time 2
cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44
+ /usr/bin/docker rm cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44
+ rustup component list --toolchain stable-x86_64-unknown-linux-gnu

• The most critical log shows that the log package cache cannot be found during compilation:

error: no matching package named `log` found
location searched: registry `crates-io`
required by package `playground v0.1.0 (/cross/project)`
As a reminder, you're using offline mode (--offline) which can sometimes cause surprising resolution failures, if this error is too confusing you may wish to retry without the offline flag.

• But in fact it can be list before compilation:

$ ls /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/log-0.4.21
CHANGELOG.md  Cargo.toml  Cargo.toml.orig  LICENSE-APACHE  LICENSE-MIT  README.md  benches  src  triagebot.toml

Try To Analyze This Bug

• I began to suspect that the /root/.cargo directory was not copied into the volume. I tried executing some command after cross build executing /usr/bin/docker cp -a /root/.cargo cross-stable-x86_64-unknown-linux-gnu-*-unknown-linux-gnu-playground-*:/cross/cargo, to see what the directory looks like, and use the following command：

$ docker ps | egrep -v "CONTAINER|test_offline:latest" | awk '{ print $1}' | xargs -I {} docker exec {} ls /cross/cargo/ -alA

total 0
drwxr-xr-x 4 root root  31 Apr 28 07:49 .
drwxr-xr-x 4 root root  31 Apr 28 07:49 ..
drwxr-xr-x 4 root root 136 Apr 28 05:34 .cargo
drwxr-xr-x 2 root root   6 Apr 28 07:49 bin

• It was found that /root/.cargo was copied to the /cross/cargo/.cargo directory, causing all caches to become invalid. Since the directory was not empty, mount entered a bin directory, resulting in home:: cargo_home_with_cwd_env (x, x) cannot automatically switch to /cross/cargo/.cargo
  • home::cargo_home_with_cwd_env
• I tried many times, but couldn't get around the problem of incorrectly setting CARGO_HOME=/cargo, so I started trying to change the cross source code in my own workspace

Try To Solve This Bug

• I tried to replace this line of code, but I don't know if it will cause other problems：
  • from :https://github.com/cross-rs/cross/blob/v0.2.5/src/docker/remote.rs#L282
  • to:

        if container_path_exists(engine, container, &dst, msg_info)? {
            for entry in fs::read_dir(cargo_dir)
                .wrap_err_with(|| format!("when reading directory {:?}", cargo_dir))?
            {
                let file = entry?;
                copy_volume_files(engine, container, &file.path(), &dst, msg_info)?;
            }
        } else {
            copy_volume_files(engine, container, cargo_dir, &dst, msg_info)?;
        }

• it works!

root@267382900f9e:/workspace# CROSS_REMOTE_COPY_REGISTRY=1 CROSS_REMOTE=1 CROSS_REMOTE_COPY_CACHE=1 ./cross build --offline --target aarch64-unknown-linux-gnu
Successfully copied 19.7MB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/cargo
Successfully copied 2.05kB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/cargo
Successfully copied 1.54kB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/cargo
Successfully copied 50.6MB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/cargo
Successfully copied 2.05kB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/cargo
Successfully copied 2.05kB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/cargo
Successfully copied 1.54kB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/cargo
Successfully copied 49.1MB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/rust
Successfully copied 1.25MB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/rust
Successfully copied 13.8kB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/rust
Successfully copied 299MB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/rust
Successfully copied 855kB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/rust
Successfully copied 149MB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/rust/lib/rustlib
Successfully copied 253MB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/rust/lib/rustlib
Successfully copied 38.3MB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-playground-c4b44:/cross/project
   Compiling log v0.4.21
   Compiling playground v0.1.0 (/cross/project)
    Finished dev [unoptimized + debuginfo] target(s) in 0.64s
Successfully copied 4.67MB to /workspace
root@267382900f9e:/workspace# 

What target(s) are you cross-compiling for?

aarch64-unknown-linux-gnu

Which operating system is the host (e.g computer cross is on) running?

• macOS
• Windows
• Linux / BSD
• other OS (specify in description)

What architecture is the host?

• x86_64 / AMD64
• arm32
• arm64 (including Mac M1)

What container engine is cross using?

• docker
• podman
• other container engine (specify in description)

cross version

cross 0.2.5

Example

• I wrote a minimal reproduction myself, which is a shell script. You can try it：

#!/bin/bash

TEST_DIR=playground
TEST_DOCKER_TAG=test_offline:latest

# delete cache
sudo rm -rf $TEST_DIR

# init a cargo project with log dependency
mkdir $TEST_DIR && cd $TEST_DIR
cargo init
cargo add log

# build rust compilable image from cross based image
cat > Dockerfile << EOF
from ghcr.io/cross-rs/aarch64-unknown-linux-gnu:0.2.5

# install rustup & cargo
RUN curl --retry 3 https://sh.rustup.rs -sSf | sh -s -- -y --profile minimal --default-toolchain stable
ENV PATH=/root/.cargo/bin:$PATH

# install cross
RUN cargo install cross

# install aarch64 target
RUN rustup target add aarch64-unknown-linux-gnu

# install rust-src
RUN rustup component add rust-src

# install docker
RUN apt update \
    && apt install apt-transport-https ca-certificates curl -y \
    && install -m 0755 -d /etc/apt/keyrings \
    && curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor --yes -o /etc/apt/keyrings/docker.asc \
    && chmod a+r /etc/apt/keyrings/docker.asc \
    && echo \
        "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
        $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
        tee /etc/apt/sources.list.d/docker.list > /dev/null \
    && apt update \
    && apt install docker-ce-cli -y
EOF

docker build -t $TEST_DOCKER_TAG .

# cross build offline using `docker-in-docker` mode
docker run \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v $(pwd):/workspace \
    -w /workspace \
    -it $TEST_DOCKER_TAG /bin/bash -c "CROSS_REMOTE_COPY_REGISTRY=1 CROSS_REMOTE=1 CROSS_REMOTE_COPY_CACHE=1 cross build --offline --target aarch64-unknown-linux-gnu --verbose"

cd -

Additional information / notes

No response

[Emilgardis]

I'm not able to try this right now, but did you get the same result on the main branch?

cargo install cross --git https://github.com/cross-rs/cross

[fjnuLuis]

• I took a look at the main branch yesterday and found that it is a bit different from v0.2.5, so I haven't tried it on it. After all, it is not in the release state.
• I just tried it and a new error occurred, it seems to be more serious than before. This time it can't even find the cargo binary, does it looks like /root/.cargo/bin is not in $PATH?

root@fa8fa42ed45d:/workspace# CROSS_REMOTE_COPY_REGISTRY=1 CROSS_REMOTE=1 CROSS_REMOTE_COPY_CACHE=1 cross build --offline --target aarch64-unknown-linux-gnu --verbose
+ cargo metadata --format-version 1 --filter-platform aarch64-unknown-linux-gnu
+ rustc --print sysroot
+ /usr/bin/docker
+ /usr/bin/docker version -f '{{ .Server.Os }},,,{{ .Server.Arch }}'
+ /usr/bin/docker info -f {{.SecurityOptions}}
+ rustup toolchain list
+ rustup target list --toolchain stable-x86_64-unknown-linux-gnu
+ rustup component list --toolchain stable-x86_64-unknown-linux-gnu
+ /usr/bin/docker volume list --format {{.Name}} --filter 'name=^cross-stable-x86_64-unknown-linux-gnu'
+ /usr/bin/docker ps -a --filter 'name=cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056' --format {{.State}}
+ /usr/bin/docker run --userns host --name cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056 --rm -v /cross -v /cross/cargo/bin -d -t ghcr.io/cross-rs/aarch64-unknown-linux-gnu:main
3746297ee54ebeefb2b3432a8513859382a06ec437dd133903cc6e14841ab5b0
+ /usr/bin/docker exec cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056 sh -c 'mkdir -p '\''/cross/root'\'''
+ /usr/bin/docker cp -a /root/.xargo/. cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056:/cross/root/.xargo
Successfully copied 1.54kB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056:/cross/root/.xargo
+ /usr/bin/docker exec cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056 sh -c 'mkdir -p '\''/cross/root/.cargo'\'''
+ /usr/bin/docker cp -a /root/.cargo/. cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056:/cross/root/.cargo
Successfully copied 86.6MB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056:/cross/root/.cargo
+ /usr/bin/docker exec cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056 sh -c 'mkdir -p '\''/cross/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib'\'''
+ /usr/bin/docker cp -a /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056:/cross/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu
Successfully copied 49.1MB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056:/cross/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu
+ /usr/bin/docker cp -a /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/libexec cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056:/cross/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu
Successfully copied 1.25MB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056:/cross/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu
+ /usr/bin/docker cp -a /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/etc cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056:/cross/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu
Successfully copied 13.8kB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056:/cross/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu
+ /usr/bin/docker cp -a /root/.local/share/cross-rs/tmp/.tmpsf37tk/lib cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056:/cross/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu
Successfully copied 299MB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056:/cross/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu
+ /usr/bin/docker cp -a /root/.local/share/cross-rs/tmp/.tmpxjwsUN/lib cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056:/cross/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu
Successfully copied 855kB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056:/cross/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu
+ /usr/bin/docker cp -a /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056:/cross/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib
Successfully copied 149MB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056:/cross/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib
+ /usr/bin/docker cp -a /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056:/cross/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib
Successfully copied 253MB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056:/cross/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib
+ /usr/bin/docker exec cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056 sh -c 'mkdir -p '\''/cross/'\'''
+ /usr/bin/docker cp -a /workspace/. cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056:/cross/workspace
Successfully copied 12.3kB to cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056:/cross/workspace
+ /usr/bin/docker exec cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056 sh -c 'set -e pipefail
set -x
chown -R 0:0 /cross
prefix="/cross"

symlink_recurse() {
    for f in "${1}"/*; do
        dst=${f#"$prefix"}
        if [ -f "${dst}" ]; then
            echo "invalid: got unexpected file at ${dst}" 1>&2
            exit 1
        elif [ -d "${dst}" ]; then
            symlink_recurse "${f}"
        else
            ln -s "${f}" "${dst}"
        fi
    done
}

symlink_recurse "${prefix}"
'
+ chown -R 0:0 /cross
+ prefix=/cross
+ symlink_recurse /cross
+ dst=/cargo
+ [ -f /cargo ]
+ [ -d /cargo ]
+ ln -s /cross/cargo /cargo
+ dst=/root
+ [ -f /root ]
+ [ -d /root ]
+ symlink_recurse /cross/root
+ dst=/root/*
+ [ -f /root/* ]
+ [ -d /root/* ]
+ ln -s /cross/root/* /root/*
+ dst=/workspace
+ [ -f /workspace ]
+ [ -d /workspace ]
+ ln -s /cross/workspace /workspace
+ /usr/bin/docker exec --user 0:0 -e 'XARGO_HOME=/root/.xargo' -e 'CARGO_HOME=/root/.cargo' -e 'CROSS_RUST_SYSROOT=/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu' -e 'CARGO_TARGET_DIR=/target' -e 'CROSS_RUNNER=' -e CROSS_REMOTE_COPY_CACHE -e CROSS_REMOTE -e CROSS_REMOTE_COPY_REGISTRY -e TERM -e CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_RUNNER -e http_proxy -e CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER -e https_proxy -e 'USER=root' -e 'CROSS_RUSTC_MAJOR_VERSION=1' -e 'CROSS_RUSTC_MINOR_VERSION=77' -e 'CROSS_RUSTC_PATCH_VERSION=2' -w /workspace cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056 sh -c 'PATH="$PATH":"/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin" cargo build --offline --target aarch64-unknown-linux-gnu --verbose --target-dir target'
sh: 1: cargo: not found
+ /usr/bin/docker exec cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056 bash -c '[[ -d '\''/cross//workspace/target'\'' ]]'
+ /usr/bin/docker cp -a cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056:/workspace/target /workspace
Successfully copied 5.12kB to /workspace
+ /usr/bin/docker stop cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056 --time 2
cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056
+ /usr/bin/docker rm cross-stable-x86_64-unknown-linux-gnu-53e1a-25ef9e3d8-aarch64-unknown-linux-gnu-8af22-1714360407056
+ rustup component list --toolchain stable-x86_64-unknown-linux-gnu

[Emilgardis]

I think we should apply your suggested fix. However, are you sure you can't use CROSS_CONTAINER_IN_CONTAINER instead? Do you need remote?

[fjnuLuis]

Yes, my docker storage driver is not overlay2, and this is not a service I can control, so I can’t use CROSS_CONTAINER_IN_CONTAINER and can only copy /root/.cargo to the new docker container.