openhpc/ohpc

Why is OHPC's py3-scipy module within an MPI hierarchy?

sunghjung3 opened this issue · 8 comments

sunghjung3 commented

Please refer to https://lists.openhpc.community/g/users/topic/96868484#4871

Scipy doesn't require MPI to build, so I think it should just be under a compiler hierarchy, NOT a compiler+MPI hierarchy.

adrianreber commented

I agree, I do not see any reason why scipy is MPI dependent. We probably can change that for the upcoming 3.x release.

CC: @mghpcsim

martin-g commented

scipy depends on other components which seem to be MPI dependent (e.g. openblas, fftw, numpy). I guess this is the reason scipy to be MPI dependent too.

adrianreber commented

Good point, right, the way OpenHPC distributes fftw makes it MPI dependent.

sunghjung3 commented

Oh, I see. Thanks. I have one more related question.

Currently, OHPC's python package modules (numpy, scipy, and mpi4py) only work for the python3.6 that was preinstalled with the Base OS (Rocky8). If I install or build a newer version of python and try to import, for example, OHPC's py3-numpy, it gives an error like this:

ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  • The Python version is: Python3.11 from "/opt/ohpc/pub/libs/python3-core/3.11.1/bin/python"
  • The NumPy version is: "1.19.5"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy.core._multiarray_umath'

Since all the users in the cluster will be using the newer Python (which is at a path that is NFS mounted by the computes) instead of the old preinstalled Base OS Python, do you ever expect us to use these OHPC Python package modules? As of now, it seems to me that these are just taking up space in /opt/ohpc/pub/libs rather than being of any use. Please correct me if this is not how it was intended to be used, or if there is a work-around to this.

adrianreber commented

No, you can only use the OHPC Python packages with the Python version they have been compiled with.

As of now, it seems to me that these are just taking up space in /opt/ohpc/pub/libs rather than being of any use.

You can always uninstall the packages if you do not need them.

sunghjung3 commented

Oh, ok. I know that the OHPC developers took time and care to compile these packages from source to make them fast, and I wanted to find the best use for them instead of just removing them.

Are there future plans to include a newer Python interpreter as part of OHPC?

mghpcsim commented
adrianreber commented

Are there future plans to include a newer Python interpreter as part of OHPC?

No. There are other tools that help you managing Python installations and additional modules. We usually just build on top of the OS provided Python. For the next release that is targeting newer base OS (RHEL 9, Leap 15.4, openEuler 22.03) it will use a newer version of Python.

If you have specific Python needs then it makes sense to use a tool that can do that for you. OpenHPC gives you a starting point.