This repository contains a collection of tools that are aimed at analysis
of distributed-memory parallel applications written with MPI. The profiling
interface provided by MPI makes it possible to collect detailed information
about messaging, and it also provides a convenient place to enable other
performance tools, including program sampling via interrupts, or collection
of aggregate values for hardware counter events. The methods to build and
use these tools are described in separate directories. A brief overview is
sketched here. Please consult the README files in each directory for more
information.
================================================================================
Tool : libmpitrace.so ; directory = src
purpose : Collect and report information on MPI calls, task placement,
memory utilization, user and system time.
outputs : text files : mpi_profile.jobid.rank
optional binary program-sampling outputs : vmon.out.jobid.rank
requires : mpicc with the underlying C compiler set to gcc.
optional : Enable program sampling via the profil() routine.
Program-sampling requires GNU binutils development files.
Note : a different program sampling method using hardware counters
is preferred ... see the section on libhpmprof.so.
build : cd src
./configure (builds only the MPI wrappers)
or ./configure --with-vprof --with-binutils=/path/to/binutils
make libmpitrace.so
typical use : export LD_PRELOAD=/path/to/libmpitrace.so
mpirun --np 2048 your.exe
unset LD_PRELOAD
================================================================================
Tool : libmpihpm.so ; directory = src
purpose : Provides the same MPI information as libmpitrace.so, plus
enables collection and reporting of aggregate values for
hardware counters. By default counts are reported from
MPI_Init() to MPI_Finalize(), but one can instrument the
code with calls to HPM_Start("label"); HPM_Stop("label");
to collect counts for specific code sections.
outputs : text files : mpi_profile.jobid.rank ... MPI data
hpm_job_summary.jobid.group ... counter data
requires : mpicc with the underlying C compiler set to gcc.
PAPI include and library paths, and a suitable set of
hardware counters for your system's CPUs.
build : cd src
./configure --with-hpm=core --with-papi=/path/to/papi
or ./configure --with-hpm=uncore --with-papi=/path/to/papi
make libmpihpm.so
typical use : export LD_PRELOAD=/path/to/libmpihpm.so
mpirun --np 2048 your.exe
unset LD_PRELOAD
================================================================================
Tool : libhpmprof.so ; directory = hpmprof
purpose : Provides the same MPI information as libmpitrace.so, plus
enables interrupt-based program sampling via hardware counters.
This library is the preferred method for program sampling for
systems that enable user-level access to hardware counters.
outputs : text files : mpi_profile.jobid.rank ... MPI data
binary files : hpm_histogram.jobid.rank ... pc sampling data
requires : mpicc with the underlying C compiler set to gcc.
PAPI include and library paths with a suitable set of
hardware counters for your system's CPUs, and GNU binutils
development files.
build : cd hpmprof
./configure --with-binutils=/path/to/binutils --with-papi=/path/to/papi
make libhpmprof.so
typical use : export LD_PRELOAD=/path/to/libhpmprof.so
mpirun --np 2048 your.exe
unset LD_PRELOAD
bfdprof your.exe hpm_histogram.jobid.rank > source_profile.txt
annotate_objdump your.exe hpm_histogram.jobid.rank > asm_profile.txt
================================================================================
Tools : bfdprof and annotate_objdump ; directory = bfdprof
purpose : These tools are required to analyze outputs generated by either of
the program-sampling methods. The bfdprof utility provides function
and statement-level profile data, and the annotate_objdump utility
provides profile data at the assembly level.
outputs : text files
requires : GNU binutils development files.
build : cd bfdprof
./configure --with-binutils=/path/to/binutils
make
typical use : bfdprof your.exe hpm_histogram.jobid.rank >source_profile.txt
annotate_objdump your.exe hpm_histogram.jobid.rank > asm_profile.txt
================================================================================
Alternate builds of libmpitrace.so
directory : ctx Adds the ability to separately report MPI profile data from
different code regions. The user must annotate the source
code and mark start/stop boundaries for each code block of
interest.
directory : nvtx Adds NVIDIA nvtx range markers around entry and exit of each
MPI function for graphical display using NVIDIA's visual
profiling tools. This is intended to add insight into the
timelines for MPI calls along with GPU kernel execution.