Make devicePersistent MPI aware
greole opened this issue · 1 comments
greole commented
Once #49 fixing #45 is implemented devicePersistentFields ie devicePersistentData<gko::Array<T>>
need some way of handling parallel execution. This includes:
- For local data, storing the MPI rank to which the data corresponds.
- Additionally, some way to handle gathered/scattered global fields. If the sparsity pattern and matrix values are not stored directly as devicePersistentField and instead the CSR matrix is kept, only the initial values and rhs are devicePersistentFields.
Data | Local | Global | Location | Updated |
---|---|---|---|---|
lduCsrMapping | persistent | not needed | host | constant |
initial guess | overwrite | persistent | device | constant |
rhs | overwrite | overwrite | device | updated |
matrix values | overwrite | overwrite | both | updated |
globalIndex* | persistent | - | host | constant |
Here, overwrite indicates that fields don't need to be stored, but storing would avoid reallocation. Global data is (for now) obtained by gathering the local data only. Thus, if global data is marked to be persistent (update false) a look up in the objectRegistry is performed.
greole commented
For now most of the functionalities related to distributed execution is handled on the ginkgo side via the partition and repartitioner class.