TST: instability on bleeding-edge CI

[neutrinoceros]

For a week now, the bleeding edge CI job has been failing, but pytest is actually erroring out on collection, it's not an internal issue.
I think I've identified the issue in that we install numpy from source (currently 1.23.0.dev...) but scipy forces downgrading, which results in an ABI incompatibility when when we import matplotlib (compiled with source numpy).

The fix is probably to install SciPy from source as well