Migrate to Numpy v2.0

Question

Migrate to Numpy v2.0

Opened this issue a month ago · 14 comments

peterfpeterson commented a month ago

Maintenance for moving to numpy v2.0 (need issue links) (migration guide)

Need prototype(s) for how extensive the changes might be
Will be done on a long-running branch to integrate the changes. main will be auto-merged into the branch
numpy v2 supports python 3.9-3.12 (python supported versions)
boost v1.84 supports newer numpy boost::python docs

Answer 1 · 2024-08-28T15:47:07.000Z

I just try to have a quick look on this.

First complication: conda dependencies. Mantid depends on quasielasticbayes which depends on numpy<2.0.0.

@robertapplin, I guess you are the one I should ask if the migration of quasielasticbayes to numpy 2 is feasible (?)

Answer 2 · 2024-08-28T17:06:22.000Z

Second complication: matplotlib. The compatibility between matplotlib and Numpy must be investigated.

This page does not mention Numpy v2 (edit: this mentioned page describes only the minimal required version of Numpy)

Even with the last penultimate version of matplotlib (3.9.1):

>>> import matplotlib

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.1.0 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

AttributeError: _ARRAY_API not found

Answer 3 · 2024-08-28T17:18:44.000Z

The good news is that source code compiles with Numpy2. I noted some tests that would require a lifting, but the effort on the source code should be limited, I hope. Moreover, the migration guide is clear.

I will try to perform some tests, and keep track in this issue.

Answer 4 · 2024-08-29T08:02:05.000Z

@robertapplin, I guess you are the one I should ask if the migration of quasielasticbayes to numpy 2 is feasible (?)

Yes I can look into this during the first sprint of v6.12

Answer 5 · 2024-08-29T09:29:10.000Z

@robertapplin, I guess you are the one I should ask if the migration of quasielasticbayes to numpy 2 is feasible (?)

Yes I can look into this during the first sprint of v6.12

Thank you. So far, we are stuck by the matplotlib issue anyway !

Answer 6 · 2024-08-29T13:12:49.000Z

Do we have confirmation as to whether matplotlib supports numpy v2 yet?

Answer 7 · 2024-08-29T13:58:09.000Z

According to here, matplotlib 3.8.4 might be compatible. I will check

Answer 8 · 2024-08-29T14:24:33.000Z

matplotlib/matplotlib#26778 suggests it too

Answer 9 · 2024-08-29T14:28:37.000Z

Indeed. Test feedback: the minimal matplotlib version you must install with numpy 2 is 3.8.4.

With matplotlib 3.8.4, Mantid workbench can be launched. Slice viewer opens. I did not push the test further. (edit: I had to remove quasielasticbayes from Mantid dependencies)

With matplotlib 3.9.X, Mantid workbench does not even start.

I will try with automatic tests, but it will take a larger amount of time :-).

Answer 10 · 2024-08-29T14:37:00.000Z

I've tested a simple environment without mantid, with numpy 2.1.0, matplotlib 3.9.2, python 3.12.5

Answer 11 · 2024-08-30T11:47:23.000Z

Indeed. Test feedback: the minimal matplotlib version you must install with numpy 2 is 3.8.4.

With matplotlib 3.8.4, Mantid workbench can be launched. Slice viewer opens. I did not push the test further. (edit: I had to remove quasielasticbayes from Mantid dependencies)

With matplotlib 3.9.X, Mantid workbench does not even start.

I will try with automatic tests, but it will take a larger amount of time :-).

For UnitTests: With matplotlib 3.8.4, Numpy 2.1.0, python 3.10.14, and after changing np.NaN to np.nan, np.Inf to np.inf, np.mat to np.asmatrix and np.alltrue to np.all

=> I am left with 21 tests failing. 9 are GUI-related and 12 are python-related (One comes from a PyStoG -a numpy.alltrue statement-)

Answer 12 · 2024-08-30T11:50:40.000Z

So my two cents, if we want to migrate to Numpy 2

we check with @robertapplin if quasielasticbayes can be migrated during the 6.12 sprint
we migrate matplotlib version to 3.8.X or 3.9.X
from this point, there should not be any unsolvable difficulty left

Answer 13 · 2024-09-04T18:57:26.000Z

I believe quasielasticbayes (v0.2.2) should now work with Numpy 2 https://anaconda.org/conda-forge/quasielasticbayes

Answer 14 · 2024-09-12T15:26:56.000Z

I believe quasielasticbayes (v0.2.2) should now work with Numpy 2 https://anaconda.org/conda-forge/quasielasticbayes

Thanks @robertapplin !

Tested with:

python 3.10.14
numpy 2.1.1
matplotlib 3.8.4
quasielasticbayes 0.2.2

=> Mantid workbench launches
=>

FAIL: test_QSe_Run (BayesQuasiTest.BayesQuasiTest)
Test Stretched Exponential fit for BayesQuasi
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/perenon/_Dev/mantid/mantill/Framework/PythonInterface/test/python/plugins/algorithms/WorkflowAlgorithms/BayesQuasiTest.py", line 79, in test_QSe_Run
    self._validate_QSe_value(result, fit_group)
  File "/home/perenon/_Dev/mantid/mantill/Framework/PythonInterface/test/python/plugins/algorithms/WorkflowAlgorithms/BayesQuasiTest.py", line 243, in _validate_QSe_value
    self.assertAlmostEqual(result.dataY(0)[0], 81.12644, delta=1e-4)
AssertionError: np.float64(81.07385699682999) != 81.12644 within 0.0001 delta (np.float64(0.05258300317001385) difference)

That will need further investigations. I will keep you up to date !