ExCALIBUR tests

Performance benchmarks and regression tests for the ExCALIBUR project.

These benchmarks are based on a similar project by StackHPC.

Note: at the moment the ExCALIBUR benchmarks are a work-in-progress.

Requirements

Spack

Note: in some HPC facilities there may be already a central Spack installation available. In principle you should be able to use that one (you only need to set the SPACK_ROOT environment variable), but you may need an up-to-date version of Spack in order to install some packages. Instructions below show you how to install Spack locally.

Spack is a package manager specifically designed for HPC facilities. Follow the official instructions to install the latest version of Spack.

In order to use Spack in ReFrame, the framework we use to run the benchmarks (see below), the directory where the spack program is installed needs to be in the PATH environment variable. This can be achieved for instance by running the commands to get shell support described in Spack documentation, which you can also add to your shell init script to do it automatically in every session. For example, if you use a shell of the family bash/zsh/sh you can add to your init script:

export SPACK_ROOT="/path/to/spack"
if [ -f "${SPACK_ROOT}/share/spack/setup-env.sh" ]; then
    source "${SPACK_ROOT}/share/spack/setup-env.sh"
fi

replacing /path/to/spack with the actual path to your Spack installation.

ReFrame requires a Spack Environment. We provide Spack environments for some of the systems that are part of the ExCALIBUR project. If you want to use a different Spack environment, set the environment variable EXCALIBUR_SPACK_ENV to the path of the directory where the environment is. If this is not set, ReFrame will try to use the environment for the current system, if known, otherwise it will automatically create a very basic environment.

ReFrame

ReFrame is a high-level framework for writing regression tests for HPC systems. For our tests we require ReFrame 3.7.3. Follow the official instructions to install this package. Note that ReFrame requires Python 3.6: in your HPC system you may need to load a specific module to have this version of Python available.

We provide a ReFrame configuration file with the settings of some systems that are part of the ExCALIBUR project. You can point ReFrame to this file by setting the RFM_CONFIG_FILE environment variable:

export RFM_CONFIG_FILE="${PWD}/reframe_config.py"

If you want to use a different ReFrame configuration file, for example because you use a different system, you can set this environment variable to the path of that file.

Note: in order to use the Spack build system in ReFrame, the spack executable must be in the PATH, also on the computing nodes of a cluster, if you want to run your benchmarks on them. Note that by default ReFrame uses

!#/bin/bash

as shebang, which would not load the user's init script. If you have added Spack to your PATH within your init script, you may want to set the RFM_USE_LOGIN_SHELL environment variable in order to make ReFrame use

!#/bin/bash -l

as shebang line, instead.

Extra Python modules

The benchmarks in this suite will additionally need the following Python modules:

Check the recommended way to install Python modules in your system, it may be for example by using pip, or creating environments with pyenv or Conda/Anaconda. For example, see the guide for CSD3.

Usage

TODO: expand

/path/to/reframe/bin/reframe -c apps/hpgmg -r --performance-report
/path/to/reframe/bin/reframe -c apps/sombrero -r --performance-report

The provided ReFrame configuration file contains the settings for multiple systems. If you use it, the automatic detection of the system may fail, as some systems may use clashing hostnames. You can always use the flag --system NAME:PARTITION to specify the system (and optionally the partition) to use.

Contributing new systems or benchmarks

Feel free to add new benchmark apps or support new systems that are part of the ExCALIBUR benchmarking collaboration. Read CONTRIBUTING.md for more details.