Anaconda3 Docker image does not have pip
Closed this issue · 12 comments
This is likely related to #120, but I'm not familiar enough with the linux build to explain.
My CI/CD pipelines that rely on the anaconda docker image have broken because pip is no longer recognized. I've tried it on two separate repos that both rely on the image and they both fail now.
Example pipeline config block
test_pyconturb: # name the job what we like
image: continuumio/anaconda3:latest
stage: # build, test, deploy defined by default [2]
test
script: # install dev options, then code, then run tests and do coverage report
- pip install pytest pytest-cov
- pip install -e .
- python -m pytest -m "not long" --cov-report term-missing:skip-covered --cov=pyconturb --cov-config .coveragerc
tags: # only runners with this tag can do the job [3]
- pythonExample of failed pipeline output
Running with gitlab-runner 10.0.2 (a9a76a50)
on synnefo-docker-iamge (761634a5)
Using Docker executor with image continuumio/anaconda3:latest ...
Using docker image sha256:8932848158968b014b012a13b2bb172aba7d0f769277cfbdd858942194f2d205 for predefined container...
Pulling docker image continuumio/anaconda3:latest ...
Using docker image continuumio/anaconda3:latest ID=sha256:9e186aada2eeee49531aa19229cf727fbab5a0d31aa15470cdf7e733647411d6 for build container...
Running on runner-761634a5-project-570-concurrent-0 via snf-713...
Fetching changes...
HEAD is now at 967a398 fixing tests
From https://gitlab.windenergy.dtu.dk/rink/pyconturb
967a398..3f28676 restruct -> origin/restruct
Checking out 3f286762 as restruct...
Skipping Git submodules setup
/bin/sh: eval: line 66: pip: not found
$ pip install pytest pytest-cov
ERROR: Job failed: exit code 127
I'll need more info about what your ci/cd is doing to provide a better answer but pip seems to work for me. For example:
Startup docker container
$ docker run -it continuumio/anaconda3:latest
Double check I have the conda I think I have - yep
(base) 36454543f5e3:/$ conda info
active environment : base
active env location : /opt/conda
shell level : 1
user config file : /home/anaconda/.condarc
populated config files :
conda version : 4.6.11
conda-build version : 3.17.8
python version : 3.7.3.final.0
base environment : /opt/conda (writable)
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/free/linux-64
https://repo.anaconda.com/pkgs/free/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /opt/conda/pkgs
/home/anaconda/.conda/pkgs
envs directories : /opt/conda/envs
/home/anaconda/.conda/envs
platform : linux-64
user-agent : conda/4.6.11 requests/2.21.0 CPython/3.7.3 Linux/4.9.125-linuxkit alpine/3.9.2 glibc/2.28
UID:GID : 10151:101
netrc file : None
offline mode : False
Check if I have pip - yep
(base) 36454543f5e3:/$ which -a pip
/opt/conda/bin/pip
Check if I can install stuff with pip - yep
(base) 5fe104b4b70e:/$ pip install pytest pytest-cov
Requirement already satisfied: pytest in /opt/conda/lib/python3.7/site-packages (4.3.1)
Collecting pytest-cov
Downloading https://files.pythonhosted.org/packages/7d/b5/92f32674ab954f80499ac73347bfeb815545ea295439c12b0ef3ac8f0975/pytest_cov-2.6.1-py2.py3-none-any.whl
Requirement already satisfied: py>=1.5.0 in /opt/conda/lib/python3.7/site-packages (from pytest) (1.8.0)
Requirement already satisfied: six>=1.10.0 in /opt/conda/lib/python3.7/site-packages (from pytest) (1.12.0)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.7/site-packages (from pytest) (40.8.0)
Requirement already satisfied: attrs>=17.4.0 in /opt/conda/lib/python3.7/site-packages (from pytest) (19.1.0)
Requirement already satisfied: atomicwrites>=1.0 in /opt/conda/lib/python3.7/site-packages (from pytest) (1.3.0)
Requirement already satisfied: pluggy>=0.7 in /opt/conda/lib/python3.7/site-packages (from pytest) (0.9.0)
Requirement already satisfied: more-itertools>=4.0.0 in /opt/conda/lib/python3.7/site-packages (from pytest) (6.0.0)
Collecting coverage>=4.4 (from pytest-cov)
Downloading https://files.pythonhosted.org/packages/b4/6a/8e2918c56682739cf177a7358929fed427d3982cc649e859b7a2d5c7daea/coverage-4.5.3-cp37-cp37m-manylinux1_x86_64.whl (204kB)
100% |████████████████████████████████| 215kB 184kB/s
Installing collected packages: coverage, pytest-cov
Successfully installed coverage-4.5.3 pytest-cov-2.6.1
Maybe pip is returning a non-zero exit code????? - nope
(base) 5fe104b4b70e:/$ echo $?
0
Pip seems to be working fine
so, I poked around a little bit more, and if you run the docker image overriding the entrypoint, then you can get into a sticky situation
$ docker run -it continuumio/anaconda3:latest sh
/ $ which -a pip
/ $
so, my guess is that your ci/cd is overwriting the entrypoint somehow? I can't really test this since I don't have gitlab steup but maybe this is a good jumping off point for your investigation. I suspect that not setting an entrypoint, or setting it to sh --login -i should solve your issue.
eg.
$ docker run -it continuumio/anaconda3:latest sh --login -i
(base) 3b38517696e5:/$ which -a pip
/opt/conda/bin/pip
@jennirinker: The exact same failure has begun afflicting our GitLab-CI pipelines for the past several days. I also note a suspiciously similar GitLab issue submitted yesterday, entitled: Kubernetes executor ignores Docker ENTRYPOINT. (Insert squinty eyes meme here.)
@soapy1 almost certainly has the right of it. This does appear to be a GitLab-CI regression introduced by the most recent stable release of GitLab. It would be super helpful, @jennirinker, if you wouldn't mind submitting a new GitLab-CI issue on this topic or commenting in the aforementioned GitLab-CI issue.
Thanks for the generous assistance, @soapy1. While the recent Alpine switch hasn't been without its issues, this doesn't seem to be one of them. Glorious pineapples for all! 🍍 🍍 🍍
@soapy1: Wait. This might be an Anaconda-side issue after all.
By inspection, the latest Anaconda3 Dockerfile doesn't actually define an ENTRYPOINT at all. It does define a CMD, but that's absolutely not the same thing. While the two are often understandably confused, best practices in the Dockerfile community suggest that most Anaconda Dockerfiles should probably define at least an ENTRYPOINT (and possibly but not necessarily also a CMD). Specifically:
CMD-style commands are entirely optional defaults implicitly overridden whenever any arguments are passed todocker run, which probably is not what anyone wants here.ENTRYPOINT-style commands are mandatory and explicitly overridden only when the--entrypointoption is passed todocker run, which probably is what everyone wants here.
Alpine-based Anaconda Dockerfiles appear to require usage of at least sh and likely also --login -i to ensure sane startup (e.g., environment variable initialization). It thus follows that:
# Replacing this line...
CMD [ "sh", "--login", "-i" ]
# ...with this line...
ENTRYPOINT [ "sh", "--login", "-i" ]
# ...or possibly these two lines...
ENTRYPOINT [ "sh" ]
CMD [ "--login", "-i" ]
...is likely to yield end user happiness.
It would make sense that this is something on Anaconda's side, since I've done some further testing today and other images don't have this problem. I'll reopen it. cc @soapy1
EDIT: Hmm, I just recreated the issue using python's 3.6-slim image, which is based on debian (dockerfile). That file does also not have an entrypoint defined, but it ends with a cmd command.
@jennirinker could you share what other images you used that did not have this problem?
Just in case there should be a link, we encountered a problem while using a custom entrypoint as well.
Dockerfile:
FROM continuumio/anaconda3:latest
ADD environment.yml /home/root/
# The RUN statement below did not work without tweaking the PATH :(
ENV PATH /opt/conda/bin/:$PATH
RUN conda env create -n myenv -f /home/root/environment.yml
WORKDIR /home/root
# add env activation to .bashrc, sourced in entrypoint.sh
RUN echo "source activate myenv" > ~/.bashrc
ENV PATH /opt/conda/envs/env/bin:$PATH
ADD ./entrypoint.sh /
ENTRYPOINT ["/entrypoint.sh"]entrypoint.sh:
#!/bin/bash
source ~/.bashrc
exec "$@"
It fails on run, I haven't got the error right now unfortunately.
(Ignore this message if unrelated, we found workarounds reverting to the old anaconda container.)
I'm pushing a new docker image continuumio/anaconda3:test with some changes from #124
This should also resolve #119
@jennirinker when you have a sec, please try this out
Ok so, we've made some additional changes. The latest and 2019.03 images have been rebuilt to use debian (like the old images). You should be able to update safely now. We added a new tag -alpine. This has the alpine images in case anyone still wants to use them. Hopefully this will clear up issues, while still providing access to the new anaconda.
That being said, testing to see if the docker image tagged as test would still be greatly appreciated.
Running off latest, it almost works...except make is not found so I can't build my sphinx docs. If I reinstall it before running my build scripts, it works. pip and git are fine.
For test, git is not found.
Incidentally, I had similar problems with the python 3.6 slim image: neither make nor git were found and I had to reinstall them in the before_script block.
Thanks @jennirinker! I think it's acceptable for make and git to not be in those images, they aren't really conda specific things.