python environment / platform dependency issues
riebling opened this issue · 14 comments
It seems the biggest problems with python are versioning and library management; that seems to definitely be the case installing BOOM for me. I'm not sure but there may be more dependencies than just python3. This might also be due to my workstation certainly not having a clean installation of anything python. I am not using anaconda or virtualenv. 2.7.12 is the default version of python, and 3.5.2 is the default for python3.
On some linux versions there's a well-known (to those who suffer from it) bug with pip where they changed some of the internals, leading to errors like this:
pip3 install -r requirements.txt
Traceback (most recent call last):
File "/usr/bin/pip3", line 9, in <module>
from pip import main
ImportError: cannot import name 'main'
Makefile:10: recipe for target 'install' failed
make: *** [install] Error 1
Thankfully what worked, after trying several more drastic things (and so I should probably submit this in the form of a pull request) was to replace instances of:
pip install xxx
with
python3 -m pip install xxx
Having gotten around this, there were some more complaints:
Installing collected packages: websocket-client, docker-pycreds, certifi, urllib3, chardet, idna, requests, docker, python-gflags, glog, numpy, pika, pydotplus, mccabe, isort, lazy-object-proxy, typed-ast, wrapt, astroid, pylint, pymongo, attrs, atomicwrites, py, pluggy, more-itertools, pathlib2, pytest, rouge
Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/local/lib/python3.5/dist-packages/websocket_client-0.48.0.dist-info'
Consider using the `--user` option or check the permissions.
Makefile:10: recipe for target 'install' failed
make[1]: *** [install] Error 1
make[1]: Leaving directory '/usr1/er1k/BOOM'
Makefile:4: recipe for target 'all' failed
make: *** [all] Error 2
So now it looks like we should also add the --user switch to the pip install commands.
This gets me further though, to an error in testing:
ImportError while importing test module '/usr1/er1k/BOOM/tests/test_job.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_job.py:1: in <module>
from boom import Job
boom/__init__.py:7: in <module>
from .pipeline import Pipeline
boom/pipeline.py:6: in <module>
import pika
E ImportError: No module named pika
So I tried working around this by installing pika, even though there are several ways that come to mind, what worked for me was
sudo apt-get install python3-pika
Now similar process to resolve:
ImportError while importing test module '/usr1/er1k/BOOM/tests/modules/test_csv_writer.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/modules/test_csv_writer.py:1: in <module>
from boom import Job
boom/__init__.py:7: in <module>
from .pipeline import Pipeline
boom/pipeline.py:7: in <module>
import pydotplus
E ImportError: No module named pydotplus
with
sudo apt-get install python3-pydot
or maybe
python3 -m pip install pydotplus
but now I'm stuck. It looks like I do have pydotplus, but the environment that runs the test in Makefile is not finding it:
python3 -m pip install pydotplus
Requirement already satisfied: pydotplus in /usr1/er1k/.local/lib/python3.5/site-packages (2.0.2)
Requirement already satisfied: pyparsing>=2.0.1 in /usr1/er1k/.local/lib/python3.5/site-packages (from pydotplus) (2.1.10)
If I try a simple test, I can import pydotplus:
er1k@islpc22:~/BOOM$ python3
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pydotplus
>>>
So I guess the environment in which make test runs is not the same as my shell.
Sorry for the Python version problem. I don't have a very good solution for it. I use Python3 on my Mac, and the Docker base image I am using requires python3. But it also works on Python2 if you replace pip3 by pip and python3 by python. What do you think is the best way to deal with this problem? Maybe forcing users to use virtual env or Anaconda?
BTW, I have a Ubuntu 16.04.4 virtual server (AWS), and the makefile works if you replace pip3 by pip and python3 by python.
nice, I am happy to stay with python 2 :)
Let's try it and see
Oh it looks like the command I forgot to edit in Makefile to use python3 was still using python (which defaults to 2 and did not have the libraries) this all makes sense now, sorry. But also, hopefully these examples are educational to others
Did you install all dependencies? Which Linux version are you using?
I've got 16.04 (CMU's customized version). I did install rabbit.
Great experience for 'user testing' - I didn't follow the directions and typed 'make' instead of 'make install'
make install worked as advertised in the README. my accidental make ran the tests,
some of which didn't pass. But let's not look at those yet.
Ok also needed to install gflags: python3 -m pip install --user python-gflags
When running boom, the shebang line has it run default #!/usr/bin/env python but maybe that should be python3. Moving on, some exciting output!
er1k@islpc22:~/BOOM/examples/toy$ boom
W0813 16:27:27.916741 2945 boom:160] The experiment name is: 2018-08-13_16h27m27s
Traceback (most recent call last):
File "/usr0/home/er1k/.local/lib/python3.5/site-packages/docker/api/client.py", line 229, in _raise_for_status
response.raise_for_status()
File "/usr0/home/er1k/.local/lib/python3.5/site-packages/requests/models.py", line 939, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.35/containers/create?name=rabbitmq
Should I maybe have run make docker first?
It's because my default Python is Python 3. I'll deal with it today.
You can run BOOM on local mode without Docker. I'll push a new docker file today.
Yep, works better if you run make docker first! Works better if installed with python3, you run it with python3. This is, I hope, success output from running boom on the toy example:
er1k@islpc22:~/BOOM/examples/toy$ boom
W0813 16:37:13.243269 9048 boom:160] The experiment name is: 2018-08-13_16h37m13s
W0813 16:37:19.622383 9048 boom:205] Loglevel: INFO
W0813 16:37:19.622637 9048 boom:206] Loaded configuration file from: conf.yaml
W0813 16:37:19.622781 9048 boom:207] Running mode: docker
Thanks for the quick response interactive debug help!
Yes. All data is stored in data dir. If you want to access it, you have to start MongoDB by running
mongod --dbpath ./data --bind_ip 127.0.0.1
then using mongofiles to access the GridFS data
mongofiles -d boom get log.txt
I am so glad someone is trying BOOM : )
@riebling I updated BOOM, now everything is working on Python 2. Could you please take a look? Thanks!
Trying a fresh clone into my probably-still-dirty Python environment, and to quote another person on the internet, got this gem:
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-IbKnJc/pylint/
You are using pip version 8.1.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Makefile:13: recipe for target 'install' failed
make: *** [install] Error 1
Which I worked around by adding --disable-pip-version-check to the pip commands in Makefile.
Next error:
Collecting pylint (from -r requirements.txt (line 6))
Using cached https://files.pythonhosted.org/packages/04/1f/1d3929051b45c3e4015178c5fe5bbee735fb4e362e0fc4f0fbf3f68647ad/pylint-2.1.1.tar.gz
Complete output from command python setup.py egg_info:
Couldn't find index page for 'pytest-runner' (maybe misspelled?)
No local packages or download links found for pytest-runner
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-build-HYddby/pylint/setup.py", line 177, in <module>
install()
File "/tmp/pip-build-HYddby/pylint/setup.py", line 174, in install
**kwargs)
File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
_setup_distribution = dist = klass(attrs)
File "build/bdist.linux-i686/egg/setuptools/dist.py", line 260, in __init__
File "build/bdist.linux-i686/egg/setuptools/dist.py", line 284, in fetch_build_eggs
File "build/bdist.linux-i686/egg/pkg_resources.py", line 563, in resolve
File "build/bdist.linux-i686/egg/pkg_resources.py", line 799, in best_match
File "build/bdist.linux-i686/egg/pkg_resources.py", line 811, in obtain
File "build/bdist.linux-i686/egg/setuptools/dist.py", line 327, in fetch_build_egg
File "build/bdist.linux-i686/egg/setuptools/command/easy_install.py", line 434, in easy_install
File "build/bdist.linux-i686/egg/setuptools/package_index.py", line 475, in fetch_distribution
AttributeError: 'NoneType' object has no attribute 'clone'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-HYddby/pylint/
Makefile:13: recipe for target 'install' failed
make: *** [install] Error 1
This is all interesting in terms of helping a wider base of users who wish to install BOOM into generic possibly-unknown Python environments. It's good to make the install process more robust. But maybe I should be testing in a cleaner environment where we plan to run, like a standard Amazon Machine Image running approved Amazon Linux with Docker support. It's ironic because Python supports such clean and simple coding, but it's growth and popularity have led to feature creep of the kind that breaks backwards and cross-platform compatibility :-/ (I should write up a boilerplate "Why doesn't Python Just Work for Eric" :) )
Right. Maybe start a virtual machine or a clean Docker container on your local machine? I tried on my AWS Ubuntu machine, everything works fine.
I agree. Setting up environments and dependencies are the worst parts...
on a fresh 16.04 Amazon AMI image, installed:
- build-essential (for make)
- python2.7
- python2.7-dev
- python-pip
Made the edits to Makefile to disable the pip version warnings (because, got them again), and observed this upon make install:
Collecting typing; python_version < "3.5" (from astroid>=2.0.0->pylint->-r requirements.txt (line 6))
Using cached https://files.pythonhosted.org/packages/0d/4d/4e5985d075d241d686a1663fa1f88b61d544658d08c1375c7c6aac32afc3/typing-3.6.4-py2-\
none-any.whl
Collecting typed_ast; python_version < "3.7" and implementation_name == "cpython" (from astroid>=2.0.0->pylint->-r requirements.txt (line 6\
))
Using cached https://files.pythonhosted.org/packages/52/cf/2ebc7d282f026e21eed4987e42e10964a077c13cfc168b42f3573a7f178c/typed-ast-1.1.0.t\
ar.gz
Complete output from command python setup.py egg_info:
Error: typed_ast only runs on Python 3.3 and above.
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-hMDcKJ/typed-ast/
Makefile:13: recipe for target 'install' failed
make: *** [install] Error 1
so maybe we do need python 3.3 or above. Or maybe you could recommend a better AMI for me to start from (I went with one that had Ubuntu 16.04 and Docker, plus some CloudLens management tool)
What works for me in the VM, following python best practices (so I'm told by StackExchange) turns out to look like this in the Makefile:
python3 -m pip install --disable-pip-version-check -r requirements.txt
python3 setup.py sdist
python3 -m pip install --disable-pip-version-check dist/boom-0.1.tar.gz
Even in a VM that has no /usr/bin/python or python2.7 installed at all(!)
Another thing to worry about running in cloud machines: the root permission requirement for docker. So make docker didn't work, but sudo make docker did, because the VM had not been set up to work around the sudo requirements (also a best practice) by e.g. setting docker group/user memberships.
...
Successfully built boom
Installing collected packages: boom
Successfully installed boom-0.1
---> f9a76e805cc8
Removing intermediate container eb437528dc5c
Step 18/22 : RUN mkdir /data
---> Running in 98f4ab5fc480
---> 94a66807c550
Removing intermediate container 98f4ab5fc480
Step 19/22 : WORKDIR /
---> 35c480f79f86
Removing intermediate container 64bf79c97497
Step 20/22 : EXPOSE 5672
---> Running in 577bbe5e7a07
---> d17bfde0a524
Removing intermediate container 577bbe5e7a07
Step 21/22 : EXPOSE 15672
---> Running in 8aab0fd261bc
---> 3c4136dc639d
Removing intermediate container 8aab0fd261bc
Step 22/22 : EXPOSE 27017
---> Running in 0d333b7d384f
---> 4220cc2660de
Removing intermediate container 0d333b7d384f
Successfully built 4220cc2660de
ubuntu@ip-172-31-15-172:~/BOOM$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
boom/docker latest 4220cc2660de 2 minutes ago 681 MB
python 2.7-alpine 7c306adf1b3d 11 days ago 60 MB