Recommended GitLab usage

Just wondering if there’s any recommended guidelines for using cargo cross with GitLab CI. Eg whether to do docker-in-docker or to use the target architecture dockerfiles/images directly.

Hi,

I just managed to get cross running in Gitlab CI using docker executor.

There is the main points I had to pay attention to:

you have to use the current master version of cross (not the released version) since a fix has just landed recently fixing a bug preventing cross to spawn containers as there is no TTY available (#52)
in docker-in-docker environment, volumes are not as we expect them to be. The volume created by cross will not be from your script job container but from the dind service one. (Detailed explanations: https://gitlab.com/gitlab-org/gitlab-ce/issues/41227). So you have to use the fact that Gitlab CI automatically mount a volume sharing /builds/$CI_PROJECT_PATH among the services and script containers.

So, here is the Dockerfile I'm using as a base for images used by projects using Rust:

FROM debian:stretch-slim

RUN apt-get update && apt-get install -y curl build-essential

# Install rustup
ENV RUSTUP_HOME=/opt/rustup
ENV CARGO_HOME=/opt/cargo
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
ENV PATH="${PATH}:/opt/cargo/bin"

# Specify the preinstalled toolchain (the tag should match this)
ENV RUST_TOOLCHAIN="stable"

# Install toolchain
RUN rustup toolchain install "${RUST_TOOLCHAIN}"
RUN rustup default "${RUST_TOOLCHAIN}"

# Install clippy and rustfmt
RUN rustup component add clippy
RUN rustup component add rustfmt

# Install cross (and a docker client as it will be needed)
RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
RUN apt-get install -y software-properties-common apt-transport-https ca-certificates curl gnupg2
RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian stretch stable"
RUN apt-get update
RUN apt-get install -y docker-ce docker-ce-cli containerd.io
RUN cargo install --git="https://github.com/rust-embedded/cross.git" --branch="master" cross

# Modify cross tool to be able to access rustup and cargo from spawned container
RUN mv /opt/cargo/bin/cross /opt/cargo/bin/real-cross
COPY cross /opt/cargo/bin/cross
RUN chmod +x /opt/cargo/bin/cross

So, it basically just install the rust toolchain with cross, but it replace cross binary with a custom script that copy rust binaries to the shared volume of gitlab ci. Here's is the script (named cross):

#!/bin/sh

# copy cargo and rustup into /builds directory which is in a shared volume
cp -r /opt/cargo /builds/${CI_PROJECT_PATH}/shared/
cp -r /opt/rustup /builds/${CI_PROJECT_PATH}/shared/

# start real cross with modified path
RUSTUP_HOME=/builds/${CI_PROJECT_PATH}/shared/rustup CARGO_HOME=/builds/${CI_PROJECT_PATH}/shared/cargo real-cross "$@"

Then, here's part of the .gitlab-ci.yml file for a rust project:

variables:
    DOCKER_HOST: tcp://docker:2375/
    DOCKER_DRIVER: overlay2
    DOCKER_TLS_CERTDIR: ""

services:
    - docker:18.09-dind

android:
    script:
        - cross test --target arm-linux-androideabi

And this runs just fine.

Hope this helps :)

I wanted a way to do this without docker-in-docker. I've forked this repository to my GitLab and changed the Dockerfiles for the targets I need.

In each Dockerfile, I replaced at the top:

-FROM ubuntu:18.04
+ARG RUST_VERSION
+FROM rust:$RUST_VERSION

At the bottom I add, for example for arm-unknown-linux-musleabi:

RUN apt-get update && \
    apt-get install -y \
        llvm-dev \
        libclang-dev \
        libc6-dev-i386 \
        clang

RUN rustup target add arm-unknown-linux-musleabi

Note that the additional apt-get is not necessary, I use it for paho-mqtt 's bindgen feature: eclipse/paho.mqtt.rust#58

In the .gitlab-ci.yml I create Docker images:

stages:
  - build

variables:
  RUST_VERSION: '1.42'

.build: &build
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  before_script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
  script:
    - /kaniko/executor --cache=true --build-arg RUST_VERSION=$RUST_VERSION --context $CI_PROJECT_DIR/docker --dockerfile $CI_PROJECT_DIR/docker/Dockerfile.$CI_JOB_NAME --destination $CI_REGISTRY_IMAGE:rust${RUST_VERSION}-$CI_JOB_NAME

armv5te-unknown-linux-musleabi:
  <<: *build

arm-unknown-linux-musleabihf:
  <<: *build

You can now use the generated Docker image in your projects to cross-compole from x64_64 Docker hosts:

  script:
  - cargo test --target arm-unknown-linux-musleabihf
  - cargo build --target arm-unknown-linux-musleabihf --release
  - arm-linux-musleabihf-strip target/arm-unknown-linux-musleabihf/release/dipbussy

Heads up that I still cannot build using cross in a GitLab Runner because of the phenomenon found in #260. Happy to contribute additional info if I can help debug this issue.

I created #449 that makes it possible to integrate cross into GitLab CI. You can see the usage here.

I guess what the community wants is the merge of "rust" docker images for each released rust version with each "cross" docker image. It means there would be the images like this:

rust:1.47.0-x86_64-unknown-linux-gnu
rust:1.47.0-armv7-unknown-linux-gnueabihf
...

Which could be used like the following:
docker run -it --rm -v $(pwd):/project -w /project rust:1.47.0-armv7-unknown-linux-gnueabihf cargo build --target armv7-unknown-linux-gnueabihf

Does it make sense?

@avkonst, note that docker run … -v $(pwd):/project … does not work when you want to copy local file to a remote Docker daemon (see here). The Docker daeman on GitLab CI runs remotely and that's the reason why I tried to implement support for remote Docker daemon in #449.

this was just an example how it would be used like cross does currently. Of course gitlab ci would not need to do it because it will place sources within the container on the runner host. The job image spec would be a reference to the proposed image (like: rust:1.47.0-x86_64-unknown-linux-gnu).

I see the only thing this project needs is to build images from rust image not from bare debian.

@avkonst is right (comment). first I tried something like this gitlab-ci config, in hope rust is installed in cross images:

build_api:
  image: rustembedded/cross:arm-unknown-linux-gnueabi
  script:
    - build --release --target=arm-unknown-linux-gnueabi

but I got cargo command not found.

I tried to use this setup for running cross on our gitlab ci. Everything is working fine for my small test project, but production code runs into following error while compiling:

time="2022-02-09T09:34:23Z" level=error msg="error waiting for container: unexpected EOF"
read tcp 127.0.0.1:33716->127.0.0.1:2376: read: connection reset by peer

Anyone seen this before?

can you add a sample gitlab ci file

can you add a sample gitlab ci file

Sure, this would go into the Wiki, and should look something like this (copied from above):

variables:
    DOCKER_HOST: tcp://docker:2375/
    DOCKER_DRIVER: overlay2
    CROSS_REMOTE: 1
    DOCKER_TLS_CERTDIR: ""

services:
    - docker:18.09-dind

android:
    script:
        - cross test --target arm-linux-androideabi

I don't use gitlab, so if someone can provide resources for me to get acquainted with it or to confirm this works, that would be greatly appreciated. The only change is the addition of CROSS_REMOTE: 1 as a variable. Note that this needs cross installed via git right now, via cargo install cross --git https://github.com/cross-rs/cross.

@Alexhuszagh I could test it on a project we have on a private instance of gitlab that uses cross to cross-compile to several architectures.

Just to be sure, the base image for the job can be any docker image with rust, cargo, cross (from master branch), and docker client installed, or do we need something more specific ?

@Alexhuszagh I could test it on a project we have on a private instance of gitlab that uses cross to cross-compile to several architectures.

Just to be sure, the base image for the job can be any docker image with rust, cargo, cross (from master branch), and docker client installed, or do we need something more specific ?

That should be it (using rust installed via rustup), and would be greatly appreciated.

Make sure you have rustup installed, since that is what we use to determine what targets are available and download them if needed

Oh just an FYI: if you have private dependencies (those that require SSH or other access to download), make sure you copy the cargo registry with CROSS_REMOTE_COPY_REGISTRY.

Just tried now but it unfortunately failed.

I built an image with rustup, stable toolchain preinstalled, cross, and docker.

The .gitlab-ci.yml file defines the following variables:

variables:
    DOCKER_HOST: tcp://docker:2375/
    DOCKER_DRIVER: overlay2
    DOCKER_TLS_CERTDIR: ""
    CROSS_REMOTE: 1
    GIT_SUBMODULE_STRATEGY: recursive

services:
    - docker:18.09-dind

And we test our project with the following command: cross test --lib --target arm-linux-androideabi

But this fails after downloading the cross docker image for the target architecture:

$ cross test --lib --target arm-linux-androideabi
info: downloading component 'rust-std' for 'arm-linux-androideabi'
info: installing component 'rust-std' for 'arm-linux-androideabi'
Unable to find image 'ghcr.io/cross-rs/arm-linux-androideabi:main' locally
main: Pulling from cross-rs/arm-linux-androideabi
Digest: sha256:03323e5b494394573cde68a2fc20d4e473a7075f95773df51209a86c93e9d3a4
Status: Downloaded newer image for ghcr.io/cross-rs/arm-linux-androideabi:main
Error: 
   0: No such file or directory (os error 2)

I don't see the image when executing docker images from the host.
If a manually pull the image, cross does not find it either and we have the same output.

I am not an expert of Docker, we have some mismatch between client and server versions: host is running docker v20.10.16, job image is using 19.03, and tested with dind service 18.09, 19.03 and 20.10 with same result.

Could that be the cause ?

How can I provide you with more detailed information ?

supply -v to cross, so cross test -v --lib --target arm-linux-androideabi

This is what I get:

$ cross test -v --lib --target arm-linux-androideabi
+ cargo metadata --format-version 1 --filter-platform arm-linux-androideabi
+ rustc --print sysroot
+ rustup toolchain list
+ rustup target list --toolchain stable-x86_64-unknown-linux-gnu
+ rustup target add arm-linux-androideabi --toolchain stable-x86_64-unknown-linux-gnu
info: downloading component 'rust-std' for 'arm-linux-androideabi'
info: installing component 'rust-std' for 'arm-linux-androideabi'
+ /usr/bin/docker
+ /usr/bin/docker volume inspect cross-stable-x86_64-unknown-linux-gnu-92b51-fe5b13d68
+ /usr/bin/docker ps -a --filter 'name=cross-stable-x86_64-unknown-linux-gnu-92b51-fe5b13d68-arm-linux-androideabi-smartcard-7cae8' --format {{.State}}
+ /usr/bin/docker volume inspect cross-stable-x86_64-unknown-linux-gnu-92b51-fe5b13d68-arm-linux-androideabi-smartcard-7cae8
+ /usr/bin/docker volume create cross-stable-x86_64-unknown-linux-gnu-92b51-fe5b13d68-arm-linux-androideabi-smartcard-7cae8
cross-stable-x86_64-unknown-linux-gnu-92b51-fe5b13d68-arm-linux-androideabi-smartcard-7cae8
+ /usr/bin/docker run --userns host --name cross-stable-x86_64-unknown-linux-gnu-92b51-fe5b13d68-arm-linux-androideabi-smartcard-7cae8 -v cross-stable-x86_64-unknown-linux-gnu-92b51-fe5b13d68-arm-linux-androideabi-smartcard-7cae8:/cross -e 'PKG_CONFIG_ALLOW_CROSS=1' -e 'XARGO_HOME=/xargo' -e 'CARGO_HOME=/cargo' -e 'CARGO_TARGET_DIR=/target' -e 'CROSS_RUNNER=' -e 'USER=root' --security-opt 'seccomp=/builds/libraries/smartcard/target/arm-linux-androideabi/seccomp.json' -v /cross/cargo/bin -d ghcr.io/cross-rs/arm-linux-androideabi:main sh -c 'sleep infinity'
Unable to find image 'ghcr.io/cross-rs/arm-linux-androideabi:main' locally
main: Pulling from cross-rs/arm-linux-androideabi
Digest: sha256:03323e5b494394573cde68a2fc20d4e473a7075f95773df51209a86c93e9d3a4
Status: Downloaded newer image for ghcr.io/cross-rs/arm-linux-androideabi:main
bb0abe4a00811d4823ea41b1040f7ad4f202e98ad98b502840257052c52fc024
+ /usr/bin/docker exec cross-stable-x86_64-unknown-linux-gnu-92b51-fe5b13d68-arm-linux-androideabi-smartcard-7cae8 sh -c 'mkdir -p '\''/cross/cargo'\'''
+ /usr/bin/docker cp -a /opt/cargo/bin cross-stable-x86_64-unknown-linux-gnu-92b51-fe5b13d68-arm-linux-androideabi-smartcard-7cae8:/cross/cargo
+ /usr/bin/docker cp -a /opt/cargo/env cross-stable-x86_64-unknown-linux-gnu-92b51-fe5b13d68-arm-linux-androideabi-smartcard-7cae8:/cross/cargo
+ /usr/bin/docker exec cross-stable-x86_64-unknown-linux-gnu-92b51-fe5b13d68-arm-linux-androideabi-smartcard-7cae8 sh -c 'mkdir -p '\''/cross/rust/lib/rustlib'\'''
+ /usr/bin/docker cp -a /opt/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin cross-stable-x86_64-unknown-linux-gnu-92b51-fe5b13d68-arm-linux-androideabi-smartcard-7cae8:/cross/rust
+ /usr/bin/docker cp -a /opt/rustup/toolchains/stable-x86_64-unknown-linux-gnu/libexec cross-stable-x86_64-unknown-linux-gnu-92b51-fe5b13d68-arm-linux-androideabi-smartcard-7cae8:/cross/rust
+ /usr/bin/docker cp -a /opt/rustup/toolchains/stable-x86_64-unknown-linux-gnu/etc cross-stable-x86_64-unknown-linux-gnu-92b51-fe5b13d68-arm-linux-androideabi-smartcard-7cae8:/cross/rust
+ /usr/bin/docker cp -a /root/.local/share/cross-rs/tmp/.tmpxr5JHI/lib cross-stable-x86_64-unknown-linux-gnu-92b51-fe5b13d68-arm-linux-androideabi-smartcard-7cae8:/cross/rust
+ /usr/bin/docker cp -a /root/.local/share/cross-rs/tmp/.tmpuLBd0z/lib cross-stable-x86_64-unknown-linux-gnu-92b51-fe5b13d68-arm-linux-androideabi-smartcard-7cae8:/cross/rust
+ /usr/bin/docker cp -a /opt/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu cross-stable-x86_64-unknown-linux-gnu-92b51-fe5b13d68-arm-linux-androideabi-smartcard-7cae8:/cross/rust/lib/rustlib
+ /usr/bin/docker cp -a /opt/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/arm-linux-androideabi cross-stable-x86_64-unknown-linux-gnu-92b51-fe5b13d68-arm-linux-androideabi-smartcard-7cae8:/cross/rust/lib/rustlib
+ /usr/bin/docker stop cross-stable-x86_64-unknown-linux-gnu-92b51-fe5b13d68-arm-linux-androideabi-smartcard-7cae8
cross-stable-x86_64-unknown-linux-gnu-92b51-fe5b13d68-arm-linux-androideabi-smartcard-7cae8
+ /usr/bin/docker rm cross-stable-x86_64-unknown-linux-gnu-92b51-fe5b13d68-arm-linux-androideabi-smartcard-7cae8
cross-stable-x86_64-unknown-linux-gnu-92b51-fe5b13d68-arm-linux-androideabi-smartcard-7cae8
+ /usr/bin/docker volume rm cross-stable-x86_64-unknown-linux-gnu-92b51-fe5b13d68-arm-linux-androideabi-smartcard-7cae8
cross-stable-x86_64-unknown-linux-gnu-92b51-fe5b13d68-arm-linux-androideabi-smartcard-7cae8
Error: 
   0: No such file or directory (os error 2)

I tried with setting CROSS_CONTAINER_IN_CONTAINER env variable to true (our current setup for running cross in gitlab does not need it) but that still fails:

$ cross test -v --lib --target arm-linux-androideabi
+ cargo metadata --format-version 1 --filter-platform arm-linux-androideabi
+ rustc --print sysroot
+ rustup toolchain list
+ rustup target list --toolchain stable-x86_64-unknown-linux-gnu
+ rustup target add arm-linux-androideabi --toolchain stable-x86_64-unknown-linux-gnu
info: downloading component 'rust-std' for 'arm-linux-androideabi'
info: installing component 'rust-std' for 'arm-linux-androideabi'
+ /usr/bin/docker
Error: 
   0: `docker inspect runner-d924e385-project-137-concurrent-1` failed with exit status: 1
Stderr:
   Error: No such object: runner-d924e385-project-137-concurrent-1
Stdout:
   []

And indeed, no container was named runner-d924e385-project-137-concurrent-1 during the job, the names of containers have more elements: runner-d924e385-project-137-concurrent-1-39d8d66d59edf298-build-3

reopening as there's some residual issue.

Using CROSS_CONTAINER_IN_CONTAINER is not needed.

I built a sample project on gitlab.com to test using the shared runners (https://gitlab.com/ndusart/rust-test-cross). And it actually works on these: https://gitlab.com/ndusart/rust-test-cross/-/jobs/2638500505

It must be something wrong in the configuration of my specific runner. I'll try to figure out what's the problem.

It must be something wrong in the configuration of my specific runner. I'll try to figure out what's the problem.

I think so because if volume_rm returns successfully, that means that DeleteVolume is droped, which is the last thing that happens in remote::run, which is the only command in docker::run, which is the last command run prior to exiting.

Although there could be something very unusual happening in our runner. If you rely on anything that would be in a data volume, but not present on the local filesystem, that is not built in the target directory, that could be an issue. But that would likely be an issue that would occur in cross itself. If you've got any code and need help debugging, let me know.

@Alexhuszagh, thank for implementing that feature. However, I'm running into the same issue (Error: 0: No such file or directory (os error 2), see here and here).

@Alexhuszagh, thank for implementing that feature. However, I'm running into the same issue (Error: 0: No such file or directory (os error 2), see here and here).

#885 was merged which should fix this issue. Let me know if any problems still remain.

@Alexhuszagh just tested on our private instance and this works now 😃
The issue was the same, repository contained a symlink.

Thanks for the great work ! 👍

As an alternative, it is possible to directly use the containers from cross-rs to run jobs in, without using container nesting. Downside: we have to deploy the Rust toolchain ourselves, where cross would mount it into the container.
See example in https://gitlab.com/xen-project/xen-guest-agent/-/blob/main/.gitlab-ci.yml.