MRChemSoft/vampyr

`pip install` is broken

Closed this issue · 3 comments

I'm not sure how I missed it, but pip install . generates a non-functional VAMPyR package. Something as simple as:

from vampyr import vampyr1d as vp

world = vp.BoundingBox()
mra = vp.MultiResolutionAnalysis(box=world, order=5)

fails with message:

Error: /tmp/pip-req-build-8absgsfs/_skbuild/linux-x86_64-3.9/cmake-build/_deps/mrcpp-src/src/core/MWFilter.cpp: generateBlocks(), line 209: Could not open filter: 
Aborted (core dumped)

and even running:

python -m pytest --pyargs vampyr

fails.

From the error message it seems the binary files with the filters cannot be found. I can see that they are installed at $ROOT/share/MRCPP/mwfilters, where ROOT is the root of the virtual environment. I guess the logic to find the filters in MRCPP is not as bulletproof as I thought! The CMake installation is still fully functional!

I am working on this in #62

But maybe I need to modify MRCPP to fix it. Not sure yet. When VAMPyR is installed with conda, the installation tree looks exactly the same as with pip install . 🤔

Here's an extended description of the problem. As per discussions and deliberation almost 2 years ago (I believe): we search for filters into two ahead-of-time known folders: the location within the source tree and the one within the install tree. The location of these two folders is baked into the MRCPP library.

  1. When running MRCPP_DIR=/some/where/share/cmake/MRCPP python -m pip install ., the pre-built MRCPP has the correct location of the filters subfolder (share/mwfilters) in the install tree. ✅
  2. When building VAMPyR with CMake and fetching MRCPP, we always have the correct filters subfolder in the source and the install trees. ✅
  3. When building the conda package, the source tree is removed and the install tree is sandboxed. However, conda-build knows how to relocate binaries by search-and-replace of hardcoded paths in the binary with the install prefix (which is only known once one actually installs the package with conda install) ✅
  4. When running pip install without pre-build MRCPP, source and install tree are sandboxed. Everything in the install sandbox is then copied to the correct place in the virtual environment. However, there is no binary relocation and the installed libmrcpp.so points to non-existing folders for the filters. ❌

Two options:

  1. Trying to use the filesystem library in C++17, do some introspection on where libmrcpp.so is located when loading it, and compute the location of filters relative to that.
  2. Make the CTOR in MWFilter aware of an environment variable which we would set in __init__.py before doing the re-imports.

We could embed the filters into the library itself as constexpr inline Eigen matrices. With @bjorgve we tried that approach, but the compile times become ridiculously long and the results shared library quite large too.

I have tried option 1 and it's actually not possible to find the location of the libmrcpp.so when it's loaded at runtime using std::filesystem. I'll go with option 2.