installation failure -- cython?

[olakusiak]

I am trying to install the package (tried pip, from source, and with the virtual environment), it seems to be installed, but when I do the tests and get 158 errors, all looking something like this:

ImportError while importing test module '/Users/aleksandra/software/halotools/halotools/utils/tests/test_vector_utilities.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../anaconda3/envs/ht3/lib/python3.6/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
halotools/utils/__init__.py:16: in <module>
    from .conditional_percentile import sliding_conditional_percentile
halotools/utils/conditional_percentile.py:9: in <module>
    from .engines import cython_conditional_rank_kernel
halotools/utils/engines/__init__.py:1: in <module>
    from .conditional_rank_kernel import cython_conditional_rank_kernel
E   ModuleNotFoundError: No module named 'halotools.utils.engines.conditional_rank_kernel'

cython is definitely installed (Cython version 0.29.23) though.

Any idea what might be going on here?

I'm on macOS Catalina 10.15.7.

[aphearin]

Hi @olakusiak I haven't encountered this error message before but sorry this is giving you trouble. It looks like the cython extensions are not being recognized for some reason. Have you tried installing through the conda-forge channel? That usually ensures internally consistent dependencies of all relevant packages.

I just double-checked on my local machine (which is also macOS Catalina 10.15.7) and I can compile and install using cython 0.29.22. So in case using the conda-forge channel does not resolve the issue (or in case you already were using conda-forge when you tried installing into a virtualenv) then would you mind trying downgrading to cython 0.29.22 to see if that solves the problem?

[borisbolliet]

Hi,

I am encountering the same issue on my Mac.
Downgrading to Cython 0.29.22 didn't solve the issue :/ (same error message as above)

Boris

[aphearin]

Ok, thanks for confirming the problem @borisbolliet - are you also using pip or have you tried installing with conda?

[olakusiak]

thanks @aphearin for the suggestions. I tried both conda-forge and downgrading cython, and none helped

[aphearin]

Ok thanks for the diligence @olakusiak. I'm having trouble reproducing the error locally, but I'll keep trying. In the meantime, would you be able to supply a full error/build log?

[aphearin]

Ok I just tried the following environment configuration and installation test on a macOS Catalina 10.15.7:

$ conda create -n testenv python=3.7 halotools=0.7 h5py ipython jupyter matplotlib cython pytest-astropy
$ conda activate testenv
$ ipython
>>> import halotools
>>> halotools.test_installation()

When I call the installation test within the testenv environment as above, I get no errors. Would you mind trying again using this same virtual conda environment?

[olakusiak]

@aphearin that works perfectly, thank you! (I was also able to install it without any trouble with pip on a cluster)

[aphearin]

Ok, great! I wonder if maybe the problem was resolved by installing pytest-astropy? I'll update the installation docs to include that from now on. In any case, please do let me know if you encounter any other problems with v0.7 by reopening this issue and tagging me.

[borisbolliet]

Somehow I am still encountering the same error. I attach the error log:
error.log
(I am working with Ola so we can figure out together)
Thank you so much for the fast response!

[aphearin]

Hmm, @borisbolliet is it possible that you are executing the test suite from within the root halotools directory that contains the source code of halotools? I'm not sure, but I think that is what the following lines of your error log indicates:

platform darwin -- Python 3.7.10, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /Users/boris/Work/CLASS-SZ/SO-SZ/halotools, configfile: setup.cfg

If that is the case, then this would produce this error, so could you try repeating the same call to the installation test, exactly as you have done, but first cd into some other working directory that is separate from the halotools repo:

$ conda create -n testenv python=3.7 halotools=0.7 h5py ipython jupyter matplotlib cython pytest-astropy
$ conda activate testenv
$ cd /some/random/working/directory/away/from/halotools
$ ipython
>>> import halotools
>>> halotools.test_installation()

[borisbolliet]

That works ! very good

Thank you so much,
Boris

[aphearin]

Ok, glad this is resolved @borisbolliet. Just FYI, here is what was happening. This has to do with the general way the python interpreter handles your paths and not halotools per se.

Whenever you initiate an import of some module/package, the interpreter first looks in your current local directory for that package. If it does not find it, then it moves to the next location in your path sequence. In your case here, you had a local copy of halotools in your working directory, and so the interpreter looked there, but since that local copy did not have the cython extensions built (or just not self-consistently built), then you got an import error. By moving outside of halotools and trying again, the interpreter moved on down your path sequence until it found the copy of halotools that was installed to your conda directory. This version had the extensions properly built, and so you could import the package without problems.

The upshot is this: once you have installed some package into an environment, typically you should write your code and do the work that uses that package in some location besides the package source code location.

I hope this helps clarify the mysterious error message. Let me know if you encounter any other issues in getting your analysis off the ground.