OpenCMISS/iron

Handle multiple mesh/interface decompositions

Opened this issue · 2 comments

chrispbradley commented

Currently a domain decomposition is performed on a single mesh. This then allows a field to be defined over this mesh/domain and this mesh and field is contained within a single region. However, for some problems, notably fluid-structure interaction, we have multiple regions with multiple meshes. Because computations need to be performed between this meshes (e.g., computation of the interface matrices) we need to ensure that the decompositions are such that the elements are broken up with adjacent elements through the interface having the same rank. This can be achieved by formulating a super-dual graph of all the meshes and interfaces and then weighting the links so that the domain is not broken along the mesh/interface boundary. The problem is then formulating the objects to allow for this. I thus propose changing the decomposition structure to achieve this.

  1. The current decomposition object (maybe renamed meshDecomposition?) would still be defined on a mesh in a region. It would be created/finished in the current way with the exception that you would no longer set the number of domains in this object.
  2. There is a new decompositions object (maybe named regionDecomposition?). You would create this object on a (parent) region together with a work group (which would defined the number of computational ranks for the decomposition). Between create start and finish you would add in individual mesh decomposition objects. The individual mesh decompositions would need to be from either the region that the decompositions object was in or from sub-regions under the parent region. Upon finishing this object the added mesh decompositions would then be used to create the super-dual graph and this graph would be partitioned. The individual mesh decomposition objects would then computed so that they were stand alone decomposition objects (as they are now) that can be used for fields.

An example.

Old way:

decomposition = iron.Decomposition()
decomposition.CreateStart(decompositionUserNumber,mesh)
decomposition.TypeSet(iron.DecompositionTypes.CALCULATED)
decomposition.NumberOfDomainsSet(numberOfComputationalNodes)
decomposition.CreateFinish()

geometricField = iron.Field()
geometricField.CreateStart(geometricFieldUserNumber,region)
geometricField.MeshDecompositionSet(decomposition)
geometricField.CreateFinish()

New way:

meshDecomposition = iron.MeshDecomposition()
meshDecomposition.CreateStart(meshDecompositionUserNumber,mesh)
meshDecomposition.TypeSet(iron.DecompositionTypes.CALCULATED)
meshDecomposition.CreateFinish()

regionDecomposition = iron.RegionDecomposition()
regionDecomposition.CreateStart(regionDecompositionUserNumber,region,workGroup)
meshDecompositionIndex = regionDecomposition.MeshDecompositionAdd(meshDecomposition)
regionDecomposition.CreateFinish()

geometricField = iron.Field()
geometricField.CreateStart(geometricFieldUserNumber,region)
geometricField.MeshDecompositionSet(meshDecomposition)
geometricField.CreateFinish()

New way for multiple meshes:

fluidMeshDecomposition = iron.MeshDecomposition()
fluidMeshDecomposition.CreateStart(fluidMeshDecompositionUserNumber,fluidMesh)
fluidMeshDecomposition.TypeSet(iron.DecompositionTypes.CALCULATED)
fluidMeshDecomposition.CreateFinish()

solidMeshDecomposition = iron.MeshDecomposition()
solidMeshDecomposition.CreateStart(solidMeshDecompositionUserNumber,solidMesh)
solidMeshDecomposition.TypeSet(iron.DecompositionTypes.CALCULATED)
solidMeshDecomposition.CreateFinish()

interfaceMeshDecomposition = iron.MeshDecomposition()
interfaceMeshDecomposition.CreateStart(fluidMeshDecompositionUserNumber,interfaceMesh)
interfaceMeshDecomposition.TypeSet(iron.DecompositionTypes.CALCULATED)
interfaceMeshDecomposition.CreateFinish()

regionDecomposition = iron.RegionDecomposition()
regionDecomposition.CreateStart(regionDecompositionUserNumber,parentRegion,workGroup)
meshDecompositionIndex1 = regionDecomposition.MeshDecompositionAdd(fluidMeshDecomposition)
meshDecompositionIndex2 = regionDecomposition.MeshDecompositionAdd(solidMeshDecomposition)
meshDecompositionIndex3 = regionDecomposition.MeshDecompositionAdd(interfaceMeshDecomposition)
regionDecomposition.CreateFinish()

fluidGeometricField = iron.Field()
fluidGeometricField.CreateStart(fluidGeometricFieldUserNumber,fluidRegion)
fluidGeometricField.MeshDecompositionSet(fluidMeshDecomposition)
fluidGeometricField.CreateFinish()

solidGeometricField = iron.Field()
solidGeometricField.CreateStart(solidGeometricFieldUserNumber,solidRegion)
solidGeometricField.MeshDecompositionSet(solidMeshDecomposition)
solidGeometricField.CreateFinish()

interfaceGeometricField = iron.Field()
interfaceGeometricField.CreateStart(interfaceGeometricFieldUserNumber,interface)
interfaceGeometricField.MeshDecompositionSet(interfaceMeshDecomposition)
interfaceGeometricField.CreateFinish()

rchristie commented

I suggest using the term 'DomainDecomposition' rather than 'MeshDecomposition' because it tells the field where it gets its domain from. Presumably a field's domain is not restricted to being just a mesh, so maybe anticipate creating them from objects other than meshes?

Also, maybe 'RegionDecomposer' or 'DomainDecomposer' is a better name for the other object since it's role is the action of decomposing the mesh, the objects added to it store the resulting decompositions.

beauof commented

I had a student looking into this but there wasn't any outcome in terms of usable code.

Rather than building the super-dual graph, his simplified approach was:

  • compute a mapping between fluid/interface and solid/interface nodes based on the surface elements/nodes of the provided meshes
  • decompose the interface mesh using all available ranks
  • project the interface decomposition onto the fluid and solid interface boundaries
  • compute decompositions of the fluid and solid domains (using all ranks) that respect the fluid and solid interface decomposition

A few items that we came across that come to mind:

  • element-based vs. node-based decomposition
  • one might have multiple interfaces that are not connected (e.g., one fluid domain but three immersed structures, and thus, three interface domains+matrices)
  • one might want to assign the largest number of processors to the fluid domain if it's much bigger with respect to the other computational domains
  • dof mapping for decomposed interfaces is not setup, if I remember it correctly?

We should, however, be able to deal with the items listed above using the graph approach.