Error docker inspect runner-... failed with exit status: 1

Question

Error docker inspect runner-... failed with exit status: 1

shandak opened this issue 9 months ago · 10 comments

Checklist

I've looked through the issues and pull requests for similar reports

Describe your issue

A similar issue I found there #1153 but CROSS_REMOTE=1 is set and does not help.
For some reason build does not work anymore(it worked without problems ~6 months before) and I can't figure out what the issue is. Below some related verbose info and error at the end:

cross -V
cross 0.2.5 (ed713d9 2023-05-23)
[cross] warning: unable to get metadata for package
[cross] note: Falling back to `cargo` on the host.
cargo 1.70.0 (ec8a8a0ca 2023-04-25)


docker info
WARNING: API is accessible on http://0.0.0.0:2375/ without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
WARNING: No swap limit support
Client: Docker Engine - Community
 Version:    24.0.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.5
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.18.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 20.10.9
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 5b46e404f6b9f661a205e28d59c982d3634148f8
 runc version: v1.0.2-0-g52b36a2d
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-164-generic
 Operating System: Alpine Linux v3.14 (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.761GiB
 Name: fbdf80109021
 ID: WM7C:Y4ND:WKWY:3PPC:HLXQ:2XEP:Y7OJ:5RAI:6LHX:C3QB:YY3T:YPSU
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  [127](https://git.slab.pp.ua/root/analytics-cli/-/jobs/599#L127).0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine


cross build -v --target x86_64-unknown-linux-musl --release
+ cargo metadata --format-version 1 --filter-platform x86_64-unknown-linux-musl
+ rustc --print sysroot
+ /usr/bin/docker
+ /usr/bin/docker version -f '{{ .Server.Os }},,,{{ .Server.Arch }}'
+ rustup toolchain list
+ rustup target list --toolchain stable-x86_64-unknown-linux-gnu
+ rustup target add x86_64-unknown-linux-musl --toolchain stable-x86_64-unknown-linux-gnu
info: downloading component 'rust-std' for 'x86_64-unknown-linux-musl'
info: installing component 'rust-std' for 'x86_64-unknown-linux-musl'
+ /usr/bin/docker inspect runner-xqkuh56d-project-43-concurrent-0
Error: 
   0: `/usr/bin/docker inspect runner-xqkuh56d-project-43-concurrent-0` failed with exit status: 1
Location:
   /rustc/90c54[180](https://git.slab.pp.ua/root/analytics-cli/-/jobs/591#L180)6f23a127002de5b4038be731ba1458ca/library/core/src/convert/mod.rs:727
Stderr:
   Error: No such object: runner-xqkuh56d-project-43-concurrent-0

This is the dind config with CROSS_REMOTE = 1 for gitlab ci:

  image: vshandak/cross:builder
  services:
    - name: docker:20.10.9-dind
      alias: docker
      command: ["--tls=false"]
  variables:
    DOCKER_HOST: tcp://docker:2375/
    DOCKER_DRIVER: overlay2
    DOCKER_TLS_CERTDIR: ""
    GIT_SSL_NO_VERIFY: "1"
    CROSS_REMOTE: 1
    CROSS_REMOTE_COPY_REGISTRY: 1

Dockefile body in use:

FROM debian:buster-20230502-slim

RUN apt-get update && apt-get install -y curl build-essential

# Install rustup
ENV RUSTUP_HOME=/opt/rustup
ENV CARGO_HOME=/opt/cargo
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
ENV PATH="${PATH}:/opt/cargo/bin"

ENV RUST_TOOLCHAIN="stable"

RUN rustup toolchain install "${RUST_TOOLCHAIN}"
RUN rustup default "${RUST_TOOLCHAIN}"

RUN rustup component add clippy
RUN rustup component add rustfmt

RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
RUN apt-get install -y software-properties-common apt-transport-https ca-certificates curl gnupg2
RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian buster stable"
RUN apt-get update && apt-get install -y docker-ce docker-ce-cli containerd.io libssl-dev pkg-config

# set CROSS_CONTAINER_IN_CONTAINER to inform `cross` that it is executed from within a container
ENV CROSS_CONTAINER_IN_CONTAINER=true

RUN cargo install cross --git https://github.com/cross-rs/cross

What target(s) are you cross-compiling for?

x86_64-unknown-linux-musl

Which operating system is the host (e.g computer cross is on) running?

macOS
Windows
Linux / BSD
other OS (specify in description)

What architecture is the host?

x86_64 / AMD64
arm32
arm64 (including Mac M1)

What container engine is cross using?

docker
podman
other container engine (specify in description)

cross version

cross 0.2.5 (ed713d9 2023-05-23)

Example

No response

Additional information / notes

No response

Answer 1 · 2023-10-21T09:53:50.000Z

this is on a gitlab instance right? has there maybe been an update done in the base container that could've changed something?

Answer 2 · 2023-10-21T10:31:18.000Z

it looks like cross tries to get info from not available container docker inspect runner-xqkuh56d-project-43-concurrent-0.
I called to check list of containers docker container ls it returned an empty list
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

Answer 3 · 2023-10-21T10:36:25.000Z

We get the name using $HOSTNAME here

Answer 4 · 2023-10-21T10:39:31.000Z

yeah, I saw it, but Docker does not see any of the containers inside. Seems no one value fits because nothing is available.

Answer 5 · 2023-10-21T11:00:40.000Z

can you run docker context inspect please?

Answer 6 · 2023-10-21T11:19:13.000Z

sure

docker context inspect
[
    {
        "Name": "default",
        "Metadata": {},
        "Endpoints": {
            "docker": {
                "Host": "tcp://docker:2375/",
                "SkipTLSVerify": false
            }
        },
        "TLSMaterial": {},
        "Storage": {
            "MetadataPath": "\u003cIN MEMORY\u003e",
            "TLSPath": "\u003cIN MEMORY\u003e"
        }
    }
]

Answer 7 · 2023-10-21T14:11:15.000Z

Interesting, that seems wrong

Answer 8 · 2023-10-26T07:43:19.000Z

What do you expect to see from docker context inspect instead of mine result @Emilgardis ?

Answer 9 · 2023-10-30T12:38:22.000Z

Sorry, I meant that it seems correct.

I'm not sure how to proceed with this. docker inspect is used to query about different kinds of objects in docker, and I can't remember what specific thing it's supposed to return with docker inspect $HOSTNAME