furlongm/non-mpi-cmd

non-mpi-cmd is a utility that allows you to to run non-MPI-aware commands in parallel on a cluster.

non-mpi-cmd can be used to run non-MPI-aware commands in parallel on a cluster.

For example, to run the ls command on a directory in parallel, you can use

  ./non-mpi-cmd -c /bin/ls -d /home/furlongm/non-mpi-cmd

and the (non-parallel) output will be:

$ ./non-mpi-cmd -c /bin/ls -d /home/furlongm/non-mpi-cmd
[MASTER] Running command: /bin/ls
[MASTER] Processing 5 files
[MASTER] Running with 1 processes on tango.vpac.org
[MASTER] tango.vpac.org: non-mpi-cmd.o
[MASTER] tango.vpac.org: non-mpi-cmd.c
[MASTER] tango.vpac.org: non-mpi-cmd
[MASTER] tango.vpac.org: README
[MASTER] tango.vpac.org: Makefile
[MASTER] Done.

When submitted as part of a job on a cluster, you can use non-mpi-cmd
to parallelise commands that normally don't run in parallel, e.g. text or
image processing tools that operate on one file at a time.

A sample pbs script for submitting such a job:

#!/bin/bash
#PBS -N europarl
#PBS -l nodes=32
#PBS -l pmem=200mb
#PBS -m ae
#PBS -l walltime=0:40:00
module load libxslt
cd $PBS_O_WORKDIR
mpiexec --mca mpi_warn_on_fork 0 \
/home/furlongm/bin/non-mpi-cmd -c /home/furlongm/bin/europarl-parser.sh -d .

Note that the full path to the binary or script is required, feel free to
submit a patch that searches the current PATH environment. If you need to
submit arguments to the command, enclose the entire command in quotes:

./non-mpi-cmd -c "/bin/ls -l" -d /home/furlongm/non-mpi-cmd

Adding "--mca mpi_warn_on_fork 0" to your mpiexec commands will avoid mpi
errors about forking.

To compile, type make. You make need to "module load openmpi-gcc" first.