LLNL/Caliper

mpi-trace analysis

twang15 opened this issue · 2 comments

twang15 commented

I have a MPI program P, instrumented with Caliper.
W/ CALI_CONFIG_PROFILE=mpi-trace, P produces #MPI rank .cali files.

Since the file names are randomized, I want to know how to determine which one should be used in order to determine caliper timing of different hot loops within P, and how to analyze these files?

Thanks a lot!

daboehme commented

Hi @twang15 ,

First, make sure to link the target program with libcaliper-mpiwrap.so in addition to libcaliper.so when using the mpi-trace profile. Then, the resulting .cali files should contain an mpi.rank attribute with the MPI rank for each process. With this, you can run cali-query on all files, and sort the output by rank, e.g.:

cali-query *.cali -t --sort-by mpi.rank

Alternatively, you can generate a file name based on the MPI rank with CALI_RECORDER_FILENAME. For example, CALI_RECORDER_FILENAME=%mpi.rank%.cali will create the files 0.cali, 1.cali, ..., etc. for each MPI rank.

daboehme commented

Closing this - question answered, no Caliper action items.