hpsim/OGL

Make devicePersistent MPI aware

greole opened this issue · 1 comments

Once #49 fixing #45 is implemented devicePersistentFields ie devicePersistentData<gko::Array<T>> need some way of handling parallel execution. This includes:

  1. For local data, storing the MPI rank to which the data corresponds.
  2. Additionally, some way to handle gathered/scattered global fields. If the sparsity pattern and matrix values are not stored directly as devicePersistentField and instead the CSR matrix is kept, only the initial values and rhs are devicePersistentFields.
Data Local Global Location Updated
lduCsrMapping persistent not needed host constant
initial guess overwrite persistent device constant
rhs overwrite overwrite device updated
matrix values overwrite overwrite both updated
globalIndex* persistent - host constant

Here, overwrite indicates that fields don't need to be stored, but storing would avoid reallocation. Global data is (for now) obtained by gathering the local data only. Thus, if global data is marked to be persistent (update false) a look up in the objectRegistry is performed.

For now most of the functionalities related to distributed execution is handled on the ginkgo side via the partition and repartitioner class.