Docker Build Fails Due to Missing libnccl2 Version for CUDA 10.0
Closed this issue · 1 comments
zebin-huang commented
I'm experiencing an issue during Docker build for a ROS Melodic project with NVIDIA support.
Error Message:
E: Version '2.6.4-1+cuda10.0' for 'libnccl2' was not found
The relevant section of the Dockerfile is as follows:
zebin@Procuda:~/projects/ros-docker-gui$ make nvidia_ros_melodic_cuda10_cudnn7
docker build -t turlucode/ros-melodic:nvidia -f nvidia/melodic/base/Dockerfile .
[+] Building 0.6s (37/37) FINISHED
=> [internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 3.82kB 0.0s
=> [internal] load metadata for docker.io/nvidia/opengl:1.2-glvnd-runtime-ubuntu18.04 0.4s
=> [internal] load build context 0.0s
=> => transferring context: 170B 0.0s
=> [ 1/32] FROM docker.io/nvidia/opengl:1.2-glvnd-runtime-ubuntu18.04@sha256:b16525d9eb9fa4d90b5cf20c59059cbb27b19bd8451731dead519827f0dc2961 0.0s
=> CACHED [ 2/32] RUN apt-get update && apt-get install -y locales lsb-release mesa-utils git subversion nano terminator xterm wget curl htop libssl-dev build-essential dbus-x11 software-properties-common gdb 0.0s
=> CACHED [ 3/32] RUN git clone https://gitlab.kitware.com/cmake/cmake.git && cd cmake && git checkout tags/v3.22.1 && ./bootstrap --parallel=8 && make -j8 && make install && cd .. && rm -rf cmake 0.0s
=> CACHED [ 4/32] RUN apt-get update && apt-get install -y automake autoconf pkg-config libevent-dev libncurses5-dev bison && apt-get clean && rm -rf /var/lib/apt/lists/* 0.0s
=> CACHED [ 5/32] RUN git clone https://github.com/tmux/tmux.git && cd tmux && git checkout tags/3.2 && ls -la && sh autogen.sh && ./configure && make -j8 && make install 0.0s
=> CACHED [ 6/32] RUN apt-add-repository universe 0.0s
=> CACHED [ 7/32] RUN apt-get update && apt-get install -y python-pip python build-essential && apt-get clean && rm -rf /var/lib/apt/lists/* 0.0s
=> CACHED [ 8/32] RUN /usr/bin/yes | pip install --upgrade "pip < 21.0" 0.0s
=> CACHED [ 9/32] RUN /usr/bin/yes | pip install --upgrade virtualenv 0.0s
=> CACHED [10/32] RUN /usr/bin/yes | pip install --upgrade paramiko 0.0s
=> CACHED [11/32] RUN /usr/bin/yes | pip install --ignore-installed --upgrade numpy protobuf 0.0s
=> CACHED [12/32] RUN locale-gen en_US.UTF-8 0.0s
=> CACHED [13/32] RUN apt-get update && apt-get install -y zsh && apt-get clean && rm -rf /var/lib/apt/lists/* 0.0s
=> CACHED [14/32] RUN wget https://github.com/robbyrussell/oh-my-zsh/raw/master/tools/install.sh -O - | zsh || true 0.0s
=> CACHED [15/32] RUN chsh -s /usr/bin/zsh root 0.0s
=> CACHED [16/32] RUN git clone https://github.com/sindresorhus/pure /root/.oh-my-zsh/custom/pure 0.0s
=> CACHED [17/32] RUN ln -s /root/.oh-my-zsh/custom/pure/pure.zsh-theme /root/.oh-my-zsh/custom/ 0.0s
=> CACHED [18/32] RUN ln -s /root/.oh-my-zsh/custom/pure/async.zsh /root/.oh-my-zsh/custom/ 0.0s
=> CACHED [19/32] RUN sed -i -e 's/robbyrussell/refined/g' /root/.zshrc 0.0s
=> CACHED [20/32] RUN sed -i '/plugins=(/c\plugins=(git git-flow adb pyenv tmux)' /root/.zshrc 0.0s
=> CACHED [21/32] RUN mkdir -p /root/.config/terminator/ 0.0s
=> CACHED [22/32] COPY assets/terminator_config /root/.config/terminator/config 0.0s
=> CACHED [23/32] COPY assets/terminator_background.png /root/.config/terminator/background.png 0.0s
=> CACHED [24/32] RUN sh -c 'echo "deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list' 0.0s
=> CACHED [25/32] RUN apt-key adv --keyserver 'hkp://keyserver.ubuntu.com:80' --recv-key C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654 0.0s
=> CACHED [26/32] RUN apt-get update && apt-get install -y --allow-downgrades --allow-remove-essential --allow-change-held-packages libpcap-dev libopenblas-dev gstreamer1.0-tools libgstreamer1.0-dev libgstream 0.0s
=> CACHED [27/32] RUN rosdep init && rosdep update 0.0s
=> CACHED [28/32] RUN echo "source /opt/ros/melodic/setup.bash" >> /root/.bashrc 0.0s
=> CACHED [29/32] RUN echo "export ROSLAUNCH_SSH_UNKNOWN=1" >> /root/.bashrc 0.0s
=> CACHED [30/32] RUN echo "source /opt/ros/melodic/setup.zsh" >> /root/.zshrc 0.0s
=> CACHED [31/32] RUN echo "export ROSLAUNCH_SSH_UNKNOWN=1" >> /root/.zshrc 0.0s
=> CACHED [32/32] COPY assets/entrypoint_setup.sh / 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:90cb32bc7a195b11804d9229284e349a055f86ff1c91f71d52ec3dffd9c38e2d 0.0s
=> => naming to docker.io/turlucode/ros-melodic:nvidia 0.0s
Docker Image: turlucode/ros-melodic:nvidia
docker build -t turlucode/ros-melodic:cuda10 -f nvidia/melodic/cuda10/Dockerfile .
[+] Building 12.1s (7/10)
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 2.83kB 0.0s
=> [internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/turlucode/ros-melodic:nvidia 0.0s
=> [1/7] FROM docker.io/turlucode/ros-melodic:nvidia 0.0s
=> CACHED [2/7] RUN wget -nv https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb && dpkg -i cuda-keyring_1.0-1_all.deb && rm cuda-keyring_1.0-1_all.deb 0.0s
=> CACHED [3/7] RUN apt-get update && apt-get install -y --no-install-recommends cuda-cudart-10-0=10.0.130-1 cuda-compat-10-0 && ln -s cuda-10.0 /usr/local/cuda && rm -rf /var/lib/apt/l 0.0s
=> ERROR [4/7] RUN apt-get update && apt-get install -y --no-install-recommends cuda-libraries-10-0=10.0.130-1 cuda-npp-10-0=10.0.130-1 cuda-nvtx-10-0=10.0.130-1 cuda-cublas-10-0=10.0.130-1 12.0s
------
> [4/7] RUN apt-get update && apt-get install -y --no-install-recommends cuda-libraries-10-0=10.0.130-1 cuda-npp-10-0=10.0.130-1 cuda-nvtx-10-0=10.0.130-1 cuda-cublas-10-0=10.0.130-1 libnccl2=2.6.4-1+cuda10.0 && apt-mark hold libnccl2 && rm -rf /var/lib/apt/lists/*:
#0 0.536 Get:1 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]
#0 0.561 Get:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease [1,581 B]
#0 0.610 Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
#0 0.626 Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [83.3 kB]
#0 0.673 Get:5 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 Packages [1,336 kB]
#0 0.674 Get:6 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
#0 0.703 Get:7 http://packages.ros.org/ros/ubuntu bionic InRelease [4,680 B]
#0 0.840 Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages [1,344 kB]
#0 1.081 Get:9 http://archive.ubuntu.com/ubuntu bionic/restricted i386 Packages [13.5 kB]
#0 1.083 Get:10 http://archive.ubuntu.com/ubuntu bionic/universe i386 Packages [11.3 MB]
#0 1.368 Get:11 http://packages.ros.org/ros/ubuntu bionic/main amd64 Packages [809 kB]
#0 1.495 Get:12 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [6,598 B]
#0 1.581 Get:13 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [1,688 kB]
#0 2.181 Get:14 http://packages.ros.org/ros/ubuntu bionic/main i386 Packages [28.8 kB]
#0 2.202 Get:15 http://security.ubuntu.com/ubuntu bionic-security/restricted i386 Packages [40.2 kB]
#0 2.213 Get:16 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [1,713 kB]
#0 2.755 Get:17 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [3,373 kB]
#0 2.870 Get:18 http://archive.ubuntu.com/ubuntu bionic/main i386 Packages [1,328 kB]
#0 3.068 Get:19 http://archive.ubuntu.com/ubuntu bionic/multiverse i386 Packages [177 kB]
#0 3.095 Get:20 http://archive.ubuntu.com/ubuntu bionic/multiverse amd64 Packages [186 kB]
#0 3.122 Get:21 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages [11.3 MB]
#0 3.813 Get:22 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [1,354 kB]
#0 4.235 Get:23 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [23.8 kB]
#0 4.241 Get:24 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,637 kB]
#0 4.910 Get:25 http://archive.ubuntu.com/ubuntu bionic/restricted amd64 Packages [13.5 kB]
#0 4.935 Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [50.5 kB]
#0 4.976 Get:27 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [30.8 kB]
#0 4.980 Get:28 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [2,079 kB]
#0 5.265 Get:29 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [12.7 kB]
#0 5.266 Get:30 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [2,097 kB]
#0 5.564 Get:31 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2,411 kB]
#0 5.903 Get:32 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [3,785 kB]
#0 6.421 Get:33 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [1,728 kB]
#0 6.661 Get:34 http://archive.ubuntu.com/ubuntu bionic-backports/main i386 Packages [64.0 kB]
#0 6.671 Get:35 http://archive.ubuntu.com/ubuntu bionic-backports/universe i386 Packages [20.6 kB]
#0 6.673 Get:36 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [20.6 kB]
#0 6.702 Get:37 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [64.0 kB]
#0 6.994 Fetched 50.6 MB in 6s (7,816 kB/s)
#0 6.994 Reading package lists...
#0 9.412 Reading package lists...
#0 11.37 Building dependency tree...
#0 11.71 Reading state information...
#0 11.75 E: Version '2.6.4-1+cuda10.0' for 'libnccl2' was not found
------
Dockerfile:41
--------------------
40 |
41 | >>> RUN apt-get update && apt-get install -y --no-install-recommends \
42 | >>> cuda-libraries-$CUDA_PKG_VERSION \
43 | >>> cuda-npp-$CUDA_PKG_VERSION \
44 | >>> cuda-nvtx-$CUDA_PKG_VERSION \
45 | >>> cuda-cublas-10-0=10.0.130-1 \
46 | >>> libnccl2=$NCCL_VERSION-1+cuda10.0 \
47 | >>> && apt-mark hold libnccl2 \
48 | >>> && rm -rf /var/lib/apt/lists/*
49 |
--------------------
ERROR: failed to solve: process "/bin/sh -c apt-get update && apt-get install -y --no-install-recommends cuda-libraries-$CUDA_PKG_VERSION cuda-npp-$CUDA_PKG_VERSION cuda-nvtx-$CUDA_PKG_VERSION cuda-cublas-10-0=10.0.130-1 libnccl2=$NCCL_VERSION-1+cuda10.0 && apt-mark hold libnccl2 && rm -rf /var/lib/apt/lists/*" did not complete successfully: exit code: 100
make: *** [Makefile:127: nvidia_ros_melodic_cuda10] Error 1
atasoglou commented
Thanks, confirmed! Both libnccl and libcudnn7 are end-of-life and NVIDIA removed them from their repo. But there is a work-around.