OpenSpeedShop test utility: test-script.py _______________________________________________________________________________ Quick start guide: (options described in main section) cd openspeedshop-test-suite ./test-script.py --create-env --oss-module <your openspeedshop module> ./test-script.py --create-prof <module options> ./test-script.py --build-tests ./test-script.py --create-baseline ./test-script.py --run-tests ./test-script.py --compare tests Note about -create-env and --create-prof commands # ---- If your computer is listed in the example_config folder # ---- (currently just pleiedes) you can use the following instructions # ---- and skip the first to test-script.py commands above cp example_config/pleiedes_env.json env.json cp example_config/pleiedes_profiles.json profiles.json ________________________________________________________________________________ The purpose of this utility is to test OpenSpeedShop on a number of sample programs and analyze the results. It supports three (3) major operations, building, runnning, and comparing. - Building (use --build-tests) calls one or more of the install scripts located in the build_scripts directory. - Running the tests can be done with either of the two commands (--create-baseline or --run-tests) the only difference between these commands is the location of the database files. These commands run all the oss tools on all the executable files in the bin directory. These should only be called after building the database directory is as follows: $install_dir/test_data/$oss_version/[baseline,results]/$date/ - Compare (use --compare-tests) this will run openss -batch on each of the database files in the results and baseline folders. Then it will look at the corresponding results and check to see if they are within acceptable variance (default 10.00%). results will be summarized and printed to the screen. By default the script will look for the most recent date folder in the results and baseline folders in the folder for the oss_version specified in environment file. You can manually specify either of these directories with: -b for baseline folder -r for results folder These paths can be absolute or relative to the test_data dir eg. -b oss_offline-2.2.2/baseline/2016-05-11_14:28:21.023070/ or use: --baseline-version To use the most recent baseline dir for a version other than what is specifed in the env file. eg. --baseline-version oss_offline-2.2.2 NEW: Pleides users see README_PLEIDES ________________________________________________________________________________ SAMPLE WORKFLOW: ________________________________________________________________________________ 1) Create test environment (Two steps) ________________________________________________________________________________ The test suite is configured via two (2) files, the env.json and profiles.json. By default, the test suite looks for the files with these names in its install directory. You can manually specify other files using the -p option for profiles and the -e option for environment flags. _______________________ 1) Step 1: Create the default environment file env.json with the command: _______________________ ./test-script.py --create-env --oss-module <your openspeedshop module> If you wish to build the tests with the intel compiler, also add the following arguments: --ompt-root <path to ompt root> This test-script.py command will create a default environment file env.json populated with some defaults. Please check that these are set to appropriate values before continuing. The options are as follows: bin_dir - Directory to store compiled tests in mpi_drivers - List of commands you use to run mpi applications, include argments e.g. 'mpi_drivers':['mpirun -np 2'] src_dir - Directory containing the source for the tests test_data_dir - Directory to store the test data input_dir - Directory with input files for the tests that require them. oss_version - The current version of openss. This needs to be manually specified until oss has a --version option acceptable_variance - Floating point value in percent. maximum factor by which baseline and results values can differ and still be allowed to pass ompt_root - Root directory of of ompt installation for libiomp, only used with intel compiler openss_module - Location of the openss module file. this is required when using a job controller only. _______________________ 1) Step 2: Create the default profile file profiles.json with the command: _______________________ Create the profiles.json file. this contains information about how to build the tests on your system, like the MPI implementation module to be used. ./test-script.py --create-prof <module options> The module options specify which modules will be used to build the tests. these are separated into 2 categories; compilers (--cc-modules), and mpi implementations (--mpi-modules). If no --cc-modules are specified, the script will assume that gcc is in your path and attempt to build with that. MPI modules must be specified. here is an example showing how to configure with gcc, intel, openmpi, and intel-mpi ./test-script.py --create-prof --cc-modules gcc intelcc --mpi-modules openmpi mpi-intel This file (default profiles.json) is a list of build configurations. each configuration in a json object with the following properties cc - The compiler, options are intel, gnu, pgi mpi_imp - The name of the mpi implementations, openmp, openmpi, mpt etc. This needs to match name in the binary files cc_module - The module for the compiler mpi_module - The module for the mpi implementation mpi_cmake_flag - This variable tells cmake which type of mpi implementation you are using, options are, -DMPT_DIR, -DOPENMPIDIR, -DMICH_DIR (maybe others as well?) cmake_flag_var - The environment variable or path you wish the mpi_cmake_flag to be set to, usually something like $MPI_ROOT or whatever is in your mpi module file. If this starts with a $ it will be treated as an environment variable, otherwise it will be treated as a path other_modules - A list of additional modules you may need to use to compile ________________________________________________________________________________ 2) Build the tests with the command ________________________________________________________________________________ ./test-script.py --build-tests This will call the build scripts in build_scripts_dir specifed by the compilers option in the env file. If something fails here it likely that your env.json or profiles.json configuration files are incorrect. ________________________________________________________________________________ 3) Create baselines for the various tests ________________________________________________________________________________ ./test-script.py --create-baseline This can take some time, results will be in test_data/baseline/$date by default ________________________________________________________________________________ 4) Create run data for the various tests ________________________________________________________________________________ ./test-script.py --run-tests This can take some time, results will be in test_data/results/$date by default ________________________________________________________________________________ 5) Compare the results to the baseline data ________________________________________________________________________________ ./test-script.py --compare-tests This tools looks in the most recent results and baseline folders by default soon to add specific folder specification. By default the tool checks that all values are within 5% of each other. this number will be an argument soon See ./test-script.py --help for more options ________________________________________________________________________________ Directory structure after building and running: ________________________________________________________________________________ openspeedshop-test-suite CMakeLists.txt cmake ... bin ... build_scripts run_build*.sh env.json src ... test-script.py test_data baseline oss_version date openss_dbs... results oss_version date openss_dbs...
OpenSpeedShop/openspeedshop-test-suite
OpenSpeedShop project test suite for testing OpenSpeedShop application results against expected results.
C