This is a fork of LULESH 2.0 that adds support for in situ processing using Catalyst 2.0 (released with ParaView 5.10+)
The codebase has the following changes:
Makefile
: updates to include new source files and add a new variableCATALYST_ROOT
which must be set to point to the install prefix / root for Catalyst install tree.CMakeLists.txt
: updates to build system (similar to Makefile changes)lulesh-init.cc
: updates to build aconduit_node
for the mesh when Catalyst is enabledlulesh-util.cc
: updates to add a command line argument-x
which can be used to pass Python scripts to execute to Catalyst.lulesh.cc
: call catalyst init/execute/finalize methods at appropriate locations in the simulation loop.script.py
: a simple sample ParaView-Catalyst analysis scriptscript-with-p-color.py
: a sample ParaView-Catalyst script with pseudocoloring (little more interesting images thanscript.py
).
Perhaps the easiest way to test this out is use the containerized version of this app. If you already have docker installed on your system, try this:
> cd <location of source from this repository>
> docker build --tag catalyzed-lulesh .
> docker run -it catalyzed-lulesh
This will run the Catalyst-instrumented LULESH using script-with-p-color.py
.
There are several overrides that one can use to customize how the application is
executed. One is referred to Docker run documentation for additional details.
Below are some useful customizations:
# to pass command line arguments, simply add args after the image name e.g.
> docker run -it catalyzed-lulesh -p -i 10 -s 64
# to mount a local directory to generate results under, use `-v`
# replace hostdir with a valid directory on your host system
> docker run -it -v <hostdir>:/opt/output:rw catalyzed-lulesh
# to use the 'stub' Catalyst implementation instead if ParaView-Catalyst
# you can use environment variables
> docker run -it -e CATALYST_IMPLEMENTATION_NAME=stub -it catalyzed-lulesh
# additional environment variables can be passed as follows:
> docker run -it -e CATALYST_IMPLEMENTATION_NAME=stub CATALYST_DEBUG=1 -it catalyzed-lulesh
# to use a custom Python script saved on the host as `/hostdir/customscript.py`
> docker run -it -v /hostdir:/opt/input catalyzed-lulesh -p -i 10 -x /opt/input/customscript.py
To build and run locally, instead of the previously described Docker-based approach, use the following steps.
First, you will need to fetch and build Catalyst from the official repository.
> mkdir ..../catalyst
> cd ..../catalyst
> git clone --branch for/paraview-5.9 https://gitlab.kitware.com/paraview/catalyst.git src
> mkdir build
> cd build
> cmake -G Ninja -DCATALYST_BUILD_TESTING:BOOL=OFF -DCMAKE_INSTALL_PREFIX=/packages/catalyst ../src
# do the build
> ninja
# do the install
> ninja install
Refer to Catalyst documentation for details on variables available.
- Edit Makefile to update variables such as
MPICXX
,CXX
,CATALYST_ROOT
to point to your build environment. CATALYST_ROOT
must be set to theCMAKE_INSTALL_PREFIX
specified when building Catalyst.- Then, use
make
to build lulesh. This will generatelulesh2.0
executable.
To run LULESH with default setup, simply use
> ./lulesh2.0
To run using MPI, if MPI was enabled in the build
> mpirun -np <ranks> ./lulesh2.0
Either of the forms will use the Catalyst stub but will do nothing consequential.
To execute with ParaView-Catalyst instead of the stub, you will need a standard
ParaView build with Python support enabled. You can also use ParaView 5.9.1
binaries. However, note that in that case, if you build LULESH with MPI enabled,
you must using MPICH
to build LULESH, since ParaView 5.9.1 binaries are
built with MPICH and MPI implementation.
> env LD_LIBRARY_PATH=/apps/ParaView-5.9.1/lib ./lulesh2.0 -x script.py -p -i 10
On my Linux system, this generates the following output
Running problem size 30^3 per domain until completion
Num processors: 1
Num threads: 48
Total number of elements: 27000
To run other sizes, use -s <integer>.
To run a fixed number of iterations, use -i <integer>.
To run a more or less balanced region set, use -b <integer>.
To change the relative costs of regions, use -c <integer>.
To print out progress, use -p
To write an output file for VisIt, use -v
To use a Catalyst script, use -x (requires Catalyst-enabled build)
See help (-h) for more options
cycle = 1, time = 3.876087e-06, dt=3.876087e-06
VisRTX 0.1.6, using devices:
0: Quadro P4000 (Total: 8.5 GB, Available: 7.9 GB)
( 3.202s) [pvbatch ] v2_internals.py:150 WARN| Module 'script' missing Catalyst 'options', will use a default options object
saving results in '<snip>/results'
cycle = 2, time = 8.527392e-06, dt=4.651305e-06
cycle = 3, time = 1.012168e-05, dt=1.594290e-06
cycle = 4, time = 1.145304e-05, dt=1.331356e-06
cycle = 5, time = 1.263989e-05, dt=1.186852e-06
cycle = 6, time = 1.374194e-05, dt=1.102046e-06
cycle = 7, time = 1.478970e-05, dt=1.047762e-06
cycle = 8, time = 1.580172e-05, dt=1.012024e-06
cycle = 9, time = 1.679057e-05, dt=9.888451e-07
cycle = 10, time = 1.776547e-05, dt=9.749005e-07
Run completed:
Problem size = 30
MPI tasks = 1
Iteration count = 10
Final Origin Energy = 7.011263e+06
Testing Plane 0 of Energy Array on rank 0:
MaxAbsDiff = 4.365575e-11
TotalAbsDiff = 4.376588e-11
MaxRelDiff = 1.406918e-14
Elapsed time = 1.86 (s)
Grind time (us/z/c) = 6.8933593 (per dom) ( 6.8933593 overall)
FOM = 145.06715 (z/s)
Note the warning message coming from ParaView-Catalyst (nothing to worry about
at this point). And the saving results in ...
message printed from
script.py
.
You will see data files and images for 10 timesteps saved in results
directory
next to the script. The script currently fails if the directory already exists
to avoid overwriting results. You can modify the script.py
, if needed.