Rework initialisation
Closed this issue · 4 comments
Recent issues on HPC systems to do with forking show how fragile our "automatic" initialisation is. There's a subtle interplay between MPI init and enabling the fork server for compilation, along with when the forked child is allowed to "import" anything.
The issues are:
- After MPI_Init we should not fork (either for compilation, or getting a version or anything else)
To address this we use Andreas Kloeckner's prefork.py to fork (before MPI_Init) a "fork server".
- During python "import", a fork()ed child may not also call "import", Python complains about import locks.
Plausibly the "prefork-everywhere" branch of PyOP2 addresses these problems. However, it's pretty fragile.
Problems are:
from mpi4py import MPI
# or
from petsc4py import PETScinitialize MPI automagically. Therefore, we need to make sure we've set up the fork server before either of these are imported.
Given that we don't have a top-level "init" method the user has to call after importing firedrake, we're in a bit of a bind. Suggestions?
Can't we just force from firedrake import * to call the prefork? Or is it the problem that that line itself is an import, so if we fork there then the fork server cannot import any more?
Exactly. Until that line completes, you're "importing"
Note that one way out of this is to move all the MPI/petsc initialisation into op2.init, which is currently called automagically for the user when they create a firedrake object.
We should then expose it in the API and document when one needs to do things by hand.
Closing as effectively a duplicate of firedrakeproject/firedrake#3820