CPL APP for OpenFOAM

1 ) Pre-requisites for compilation

A compiled version of CPL library (https://github.com/Crompulence/cpl-library) which has been added to the user's path using "source SOURCEME.sh"
A copy of OpenFOAM
The same GCC and MPICH version used to build CPL library

The following environment variables:

FOAM_CPL_SOCKET_LIBBIN = $FOAM_INST_DIR/cpl-socket/lib
LD_LIBRARY_PATH        = $LD_LIBRARY_PATH:$FOAM_CPL_LIBBIN

must be defined in order for a) the compilation to work and b) the library to be found by the ld linker. They are conveniently defined in the config file SOURCEME, which can be used as follows:

$  source SOURCEME.sh

2 ) Install

The install process assumes that you have OpenFOAM fully installed with all source code. The process then simply builds a top level solver, using all of the OpenFOAM shared libraries and CPL library's shared library, which is designed for coupled simulation.

First, change directory to CPL_APP_OPENFOAM-DEV,

cd CPL_APP_OPENFOAM-DEV

and create a file called CODE_INST_DIR which will tell the APP which OPENFOAM to install against,

echo "/path/to/openfoam/directory/" > CODE_INST_DIR

Note that the folder in this directory is expected to be called OpenFOAM-3.0.1. This is the currently supported OpenFOAM version.

Now, to build the various solvers, it may be as simple as,

$  make

N.B.: warnings from the included MPI headers may be ignored. This makes a range of OpenFOAM solvers, including

CPLIcoFOAM - A basic fluid solver for MD-CFD coupling which partially overlaps an MD simulation
CPLSediFOAM - SediFOAM (https://github.com/xiaoh/sediFoam) adapted to use the CPL library philosophy of minimal linked library
CPLCFDDEMFOAM - CFDDEM (https://www.cfdem.com/) adapted to use the CPL library philosophy of minimal linked library

3 ) License

CPL_APP_OpenFOAM is released under the GNU GPL v3 license. Details are found in the file LICENSE that is included with the release.

4 ) Directory Structure

This application repository is structured as follows:

src - source files which include the following
- CPLPstream: OpenFOAM's parallel-computing communications supports a number of different paradigms, including MPI. In order to support all of them, it wraps all communication functions in a library called "Pstream". CPL Library is based on MPI, but requires the MPI_COMM_WORLD communicator to be split into two "realm" communicators - one for MD, and the other for CFD. All subsequent operations that would have been on MPI_COMM_WORLD in the MD and CFD coupled programs must now be called on the realm communicator CPL_REALM_COMM. CPLPstream replicates OpenFOAM's MPI Pstream, where CPL_REALM_COMM has been substituted in place of MPI_COMM_WORLD.
- CPLSocket: The interface to CPL Library is implemented in a single class named CPLSocket. This directory holds its source code.
- solvers: OpenFOAM is implemented as a set of libraries, and it is up to the user to develop their own main-level applications that use these libraries. This directory contains the source code for a coupled incompressible solver CPLIcoFoam that is based on OpenFOAM's icoFoam.
examples - some input examples for various cases
test - a range of test cases run automatically on Travis CI
config - scripts to specify version of MPI to build OpenFOAM with and patches for scotch.

New folders created by building process

lib - dynamic-link library binaries are created in a new folder ./lib/.
bin - Executable solver applications are created in the a folder ./bin.

5 ) To patch a Pstream

Why do you need to do this? If you want to run two codes with the colon MPMD syntax,

mpiexec -n 4 CPLIcoFoam : -n 16 ./MD

where they have a shared MPI_COMM_WORLD, which we will call the "shared" paradigm of coupling (as opposed to the distinct paradigm where both codes are started individually and join together using the not always functional MPI_Open_port and MPI_Comm_accept style linking to create an intercommunicator between the MPI_COMM_WORLD intracommunicators of both codes). The sharing of MPI_COMM_WORLD means that any use of MPI_COMM_WORLD in any MPI communications will now cause errors or deadlock in the coupled code, so these have to be replaced with a local comm. Patching Pstream, the location where all MPI communication is contained, is the easiest way to do that for OpenFOAM. The steps are as follows (given in general terms to account for future OpenFOAM changes but specifically done for v2112 in the current patch.

The Pstream which is used can be replaced for all codes using LD_LIBRARY_PATH, it goes from version in

/home/USERNAME/codes/CFD/OpenFOAM/openfoam-OpenFOAM-v2112.220610/platforms/linux64GccDPInt32Opt/lib/

CPL_APP_OPENFOAM/lib/libPstream.so

The first step in patching requires copying all of the Pstream/mpi file from the OpenFOAM code diretory. e.g. in my case

/home/USERNAME/codes/CFD/openfoam-OpenFOAM-v2112.220610/src/Pstream/mpi/

to a folder called CPLPstream in the CPL_APP_OPENFOAM/src directory, replacing whatever files are there. You can try building this with wmake, which should add PStream to CPL_APP_OPENFOAM/lib and by changing LD_LIBRARY_PATH you should see included CPL_APP_OPENFOAM/lib/Pstream.so replacing the default Pstream.so on your OpenFOAM executables, e.g.

ldd /home/USERNAME/codes/CFD/OpenFOAM/openfoam-OpenFOAM-v2112.220610/platforms/linux64GccDPInt32Opt/bin/icoFoam

Now, you can edit this Pstream as needed to make sure the shared CPL paradigm works. Then adding the following line

extern MPI_Comm CPLRealmComm;

in the file PstreamGlobals.H and setting it to default to MPI_COMM_WORLD for default case which is nothing to do with coupling.

MPI_Comm Foam::PstreamGlobals::CPLRealmComm = MPI_COMM_WORLD;

in PstreamGlobals.C. At this stage, you can rebuild Pstream again and functionality should be identical (we have just add a new variable so far).

Next, find and replace every instance of MPI_COMM_WORLD in UPStream.C with Foam::PstreamGlobals::CPLRealmComm. You can rebuild Pstream and test again as nothing has changed, as the new Foam::PstreamGlobals::CPLRealmComm is still MPI_COMM_WORLD. Finally, in order to allow a "shared" MPI run, we need to define the Foam::PstreamGlobals::CPLRealmComm to be the value returned by CPL.init. In an example script this looks like:

#include "PstreamGlobals.H"
#include "mpi.h"
#include "cpl.h"

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

int main(int argc, char *argv[])
{

    //Define variables
    int CFD_realm = 1;
    MPI_Comm CFD_COMM;

    int flag = 0;
    int ierr = MPI_Initialized(&flag);
    if (flag == 0)
	    MPI_Init(&argc, &argv);

    //Initialise CPL library
    CPL::init(CFD_realm, CFD_COMM); 
    Foam::PstreamGlobals::CPLRealmComm = CFD_COMM;