Allow full installation using conda
Closed this issue · 16 comments
Add an installation method that creates a new conda virtualenv and then installs camoco through a conda channel instead of relying on pip/setuptools.
You want to release camoco as a conda package as well apart from pip?!
Hi @aeroaks, right now the installation step is a doozy. Maybe this has changed, but having a numpy/scipy dependency through pip takes >30 mins to install. Maybe they've fixed this problem with more recent versions of wheels, but whenever we run our CI tests, pip dominates in run time. Right now we have a pretty hacked together installation process (see #67 ).
All together our goals would be:
- Acceptable installation time (<10 mins)
- A unified build environment (preferably through setup.py and pip)
If you had any suggestions, we would be all ears.
From what I understand, If we can package Camoco as a Conda package then the process of setting up the environment eases considerably and hopefully reduces installation times.
For a unified build environment, If we are using pip and we have numpy/scipy as dependencies then how you intend to tackle their installation from multi-platform support perspective? I use Python on Windows for work and found conda environment setup easier to handle NumPy/PyQt4 dependencies.
Ease of installation would help us out considerably and I like that using conda gives us access to virtual environments out of the box. Do you have experience setting up channels? What would we need to get started?
If we could just have:
conda create -n camoco_env python=3
source activate camoco_env
conda install camoco
Or something as simple, we would be elated. Working towards windows compatibility is in our future too. I think we have a couple dependecies that dont play nice with windows and a lot of the scientific computing community doesn't really use it. But it'd be nice!
There are a few weird components in the install script. But it looks like some of those work arounds (it looks like bcolz is on conda now, etc) have been fixed since we've last updated the install script.
Great! I will work on it then and come back with updates!
I am trying to setup a conda environment and get following error when I try to import camoco as co in python REPL:
distutils.errors.DistutilsPlatformError: Unable to find vcvarsall.bat
Traceback (most recent call last):
File "C:\Users\Akshay\Miniconda3\envs\camoco_env\lib\site-packages\pyximport\pyximport.py", line 215, in load_module
inplace=build_inplace, language_level=language_level)
File "C:\Users\Akshay\Miniconda3\envs\camoco_env\lib\site-packages\pyximport\pyximport.py", line 191, in build_module
reload_support=pyxargs.reload_support)
File "C:\Users\Akshay\Miniconda3\envs\camoco_env\lib\site-packages\pyximport\pyxbuild.py", line 102, in pyx_to_dll
dist.run_commands()
File "C:\Users\Akshay\Miniconda3\envs\camoco_env\lib\distutils\dist.py", line 955, in run_commands
self.run_command(cmd)
File "C:\Users\Akshay\Miniconda3\envs\camoco_env\lib\distutils\dist.py", line 974, in run_command
cmd_obj.run()
File "C:\Users\Akshay\Miniconda3\envs\camoco_env\lib\site-packages\Cython\Distutils\old_build_ext.py", line 186, in run
_build_ext.build_ext.run(self)
File "C:\Users\Akshay\Miniconda3\envs\camoco_env\lib\distutils\command\build_ext.py", line 339, in run
self.build_extensions()
File "C:\Users\Akshay\Miniconda3\envs\camoco_env\lib\site-packages\Cython\Distutils\old_build_ext.py", line 194, in build_extensions
self.build_extension(ext)
File "C:\Users\Akshay\Miniconda3\envs\camoco_env\lib\distutils\command\build_ext.py", line 533, in build_extension
depends=ext.depends)
File "C:\Users\Akshay\Miniconda3\envs\camoco_env\lib\distutils\_msvccompiler.py", line 304, in compile
self.initialize()
File "C:\Users\Akshay\Miniconda3\envs\camoco_env\lib\distutils\_msvccompiler.py", line 197, in initialize
vc_env = _get_vc_env(plat_spec)
File "C:\Users\Akshay\Miniconda3\envs\camoco_env\lib\distutils\_msvccompiler.py", line 85, in _get_vc_env
raise DistutilsPlatformError("Unable to find vcvarsall.bat")
distutils.errors.DistutilsPlatformError: Unable to find vcvarsall.bat
Have you experienced it before?
Hi @aeroaks
Ive never run into this, but it looks like its from pyximport.
This appears to be a compiler issue. There are a few parts of Camoco that use Cython to compile critical loops so they don't take forever. Here is a reference with some workarounds.
You could try one more thing first. Did the installation process work ok? If so, there is a small piece of code in __init__.py that recomiles on the fly if the source code was changed (used for development). You could try commenting that out and that might fix your issue.
Removing the small init code also does not work. I get:
ImportError: No module named 'camoco.RefGenDist'
Referring to error with pyximport in __init__ code:
ImportError: Building module camoco.RefGenDist failed: ['distutils.errors.DistutilsPlatformError: Unable to find vcvarsall.bat\n']
I am trying to run import camoco as co in the root directory of the repo within my virtual environment.
Yeah, RefGenDist is one of the classes that uses Cython to compile some code.
Ok, how about that workaround from that Microsoft blog post? Maybe getting Visual Studio 2015 installed will allow pyximport to compile the Cython code.
If this doesn't fly, I'll try to get a windows virtual image running. We've never installed camoco on windows before. If you would be willing to document the experience, I can open another hacktoberfest issue and we could do a two-fer!
If you want I can share the current virtual environment setup that I am using.
I am working on this #68. Since all of our work seems to affect the installation process, don't you think it would be better to establish a common direction of work? 😄
I say, because I have been modifying install.sh script to make Camoco install on OSX and run the tests on TravisCI. If the package finally does not use a bash script to install, as said in #67, this work will have been in vain.
At least, establish a priority of tasks. Which are more important and should be done first.
@seraco Thanks for your comment. You are correct and I think that you will have a better oversight in the priority and setup process.
From what I understand, your upgrade for the installation process would define this task. You are welcome to suggest tasks that you want to try on Windows installations during this development? Then we can cover Windows installation as well.
Hi all,
I think @seraco prioritized these in the right order. Ideally, users should be able to use the conventional setup.py installation process (which can take a while due to numpy/scipy installation) or if they prefer to work with conda, they can install it from there. The reason things are so fragmented right now is because we needed to quickly install everything when we ran tests on TravisCI and pip was much too slow which resulted in the install.sh script.
Since most folks will only need to go through the pain of installing once, it'd be nice to do it in a cross-platform/conventional way with setup.py and support multiple OSs. It would also be nice to be able to quickly install things through conda so our CI tests dont take forever.
Anyways, this comment is much too long to just agree that @seraco probably has the order right. Once things are working with setup.py it'll be easier to move to conda.
I've resolved the internal camoco issues that were keeping this from being installed via a sane method. i.e. setup.py. Camoco can now be installed through pip which would make conda a possibility but for now I am going to be satisfied with pip install camoco.