Docker version incompatibility with GLIBC library version

Question

Docker version incompatibility with GLIBC library version

ItIsJAPO opened this issue a year ago · 5 comments

Jenkins and plugins versions report

I'm running Jenkins version 2.402 on both servers and docker engine 20 and 23. Attached is my Docker Compose.


services:
  jenkins_master:
    image: jenkins/jenkins: jdk11
    container_name: jenkins
    user: root
    environment:
      - TZ=America/Merida
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /usr/bin/docker:/usr/bin/docker
      - /mnt/jenkins_home:/var/jenkins_home
    ports:
      - "127.0.0.1:2345:2345"
      - 80:8080
      - 50000:50000

What Operating System are you using (both controller, and any agents involved in the problem)?

I am working with two servers running different versions of Docker engine, one with version 20 and the other with version 23. Both with version Ubuntu 22.04.2 LTS.

Reproduction steps

Prerequisites:

Two servers running different versions of Docker engine, one with version 20 and the other with version 23.
A Jenkins controller installed on one of the servers, running on a Linux-based operating system.

Steps:

Install Jenkins on the server with Docker version 20 using a clean installation.
Set up a new Jenkins job that will execute a Docker build and run command on the server.
Configure the Docker command to use the Docker version 20 installed on the same server where Jenkins is installed.
Run the job and verify that it executes successfully.
Install Jenkins on the server with Docker version 23 using a clean installation.
Set up a new Jenkins job with the same Docker build and run command as the previous job.
Configure the Docker command to use the Docker version 23 installed on the same server where Jenkins is installed.
Run the job and observe that it fails with the error message "/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by docker)" due to the incompatibility of the Docker version with the GLIBC library version on the server.

Expected Results

Both servers should allow the execution of the Docker image without any problem.
The execution of the docker command inside the container should work correctly on both servers, without throwing errors due to the Docker engine version or any other dependencies.
The execution of the Jenkins task that uses Docker images should not encounter any issues with Docker Engine compatibility on either server.

Actual Results

The job should execute successfully on the server with Docker version 20, using the Docker command configured to use the Docker version 20 installed on the same server where Jenkins is installed.
The job should fail with an error message on the server with Docker version 23, due to the incompatibility of the Docker version with the GLIBC library version on the server.

Anything else?

No response

Answer 1 · 2023-04-28T15:08:03.000Z

Hi @ItIsJAPO , thanks for raising the issue. It does not seem to be a bug in the Jenkins Controller image, but more a misconfiguration/misusage.

A few elements that should help you

Please take care of using the word "controller" instead of "master"
It is absolutely not recommended to execute Jenkins builds on the Jenkins controller:
- It is not safe (as the builds can access to the controller JENKINS_HOME and read/decrypt all the secrets)
- Separation of concern: Agents should have the required tools (such as the docker CLI) on the version you need, while the controller should only have what it needs. Mixing them is a recipe for failure.
- It does not scale at all
Mounting an executable binary from the host is not recommended (ref. https://docs.docker.com/storage/). It goes against all the benefits of using Docker:
- It creates a strong dependency to the host (Absolute path must be present, same user with same UID/GID must be present on both Docker container and host machine) making it non portable (better to use the host package manager instead of Docker if this is strongly required)
- It does not respect the separation of concerns of containers
- You can't be sure that the dynamic linking of the host binary are present in the Docker image (this is your case here :) )

Before suggesting one or the other direction, could you explain the requirement you have for Docker?

Is is because Jenkins controller need to spin up ephemeral agents with docker containers?
Is it because you want to use the agent { docker { ...}} directive in your pipeline?
Is it because you need docker as part of your build steps?
Another reason?

With the answer to these question, we can give you directions

Answer 2 · 2023-04-28T15:08:56.000Z

(To help you: ldd /usr/bin/docker will show you the expected system dynamic libraries required by the binary: I bet some are absent in Debian bullseye in the Jenkins controller official image)

Answer 3 · 2023-04-28T15:18:46.000Z

I completely understand. I wanted to make it as simple as possible to replicate the error. What I am trying to do on the slave is to create the image, so I only try to run the build, tag, and push commands. When I run ldd /usr/bin/docker on the host, it shows me the following output:

Answer 4 · 2023-04-28T16:40:40.000Z

With the latest image:

$ docker run --rm --entrypoint='' jenkins/jenkins:jdk11 ls -l /lib/x86_64-linux-gnu/libc.so.6
lrwxrwxrwx 1 root root 12 Oct 14  2022 /lib/x86_64-linux-gnu/libc.so.6 -> libc-2.31.so

This is the glibc in Debian bullseye (the base image of the Jenkins controller) and as per https://packages.debian.org/bullseye/libc6-amd64, there is not available 2.32 glibc.

From there, based on your answer, you could use one of the followings:

Install the Docker CLI yourself instead of bind mounting it:
- https://docs.docker.com/engine/install/debian/ (looks only for the package docker-ce-cli and eventually docker-buildx-plugin docker-compose-plugin in addition to gain rich features)
- Install only the client binary (https://docs.docker.com/engine/install/binaries/#install-daemon-and-client-binaries-on-linux)
Better: define an agent with the exact Docker CLI you want (and the same user UID to allows read/write on the /var/run/docker.sock)
Define a "Docker in Docker" agent (https://hub.docker.com/_/docker) to isolate Docker Engine instances
Define an agent directly on the host so it would have direct access to the Docker Engine

My recommendation would be to run Jenkins on another machine than one with the Docker Engine you want to use for your builds: if you don't, it means that anyone able to execute a build could get a root access to the host machine.
(There are countermeasure such as https://docs.docker.com/engine/security/userns-remap/ but it's a lot of pain compared to setting up another machine and be done with it).

Answer 5 · 2023-05-15T07:00:14.000Z

Hello @ItIsJAPO , since this issue is not related to a problem in the image and not a feature request, I'm going to close it.

Feel free to reopen if you have a specific bug or feature in mind.

If you need further help, you might be interested by opening a discussion thread in the community formums at community.jenkins.io (broader audience).