SYCL 2020 Conformance Test Suite
This is the SYCL Conformance Test Suite for the Khronos Group SYCL standard.
For more information: https://www.khronos.org/sycl
Introduction
This conformance test suite is divided into two parts, a binary executable containing all of the tests and a python user interface. It is intended that the user only runs the test suite via the Python front-end.
The test executable must be compiled before running the test suite, a step which requires CMake. Executables are produced for each category and one 'fat' executable that contains the tests from all categories.
Compilation and execution prerequisites
- Python 3.7
- CMake 3.15
- A conformant implementation of OpenCL
- An implementation of SYCL - Note: the SYCL implementation may require additional dependencies
The SYCL conformance test suite assumes that an underlying OpenCL implementation has passed the Khronos OpenCL CTS.
Building on Linux
Checkout the SYCL test suite repository.
Run CMake - point the source directory to the root of the
sycl-cts
folder - configure the generator to use makefiles or ninja - set the build directory to abuild
folder inside thesycl-cts
folder- set the CMake parameters
Run make inside the
build
folder
After compilation, the test executables will be placed in the
sycl-cts
build/bin
directory. The test_all
executable
contains all tests in the suite.
You can also compile each tests separately by giving target to the make command. For instance:
$ make test_header
Building on Windows
Checkout the SYCL test suite repository.
Run CMake - point the source directory to the root of the
sycl-cts
folder - select "Visual Studio 14 2015 Win64" generator - set the build directory to abuild
folder inside thesycl-cts
folder - set the CMake parametersOpen the generated solution in Visual Studio 2015 and rebuild all
After compilation, the test executables will be placed in the
sycl-cts
build/bin
directory. Thetest_all
executable contains all tests in the suite.
CMake flags
When configuring CMake, it is possible to use these flags:
COMPUTECPP_INSTALL_DIR
- Required only if your SYCL implementation is ComputeCpp.
SYCL_CTS_EXCLUDE_TEST_CATEGORIES
- Optional file specifying a list of test categories to be excluded from the build.
SYCL_CTS_ENABLE_FULL_CONFORMANCE
- Enable extensive coverage with huge compilation and execution time. This mode is switched off by default. Should be swithed on for conformance.
SYCL_CTS_VERBOSE_LOG
- Enable debug-level logs with the possibly oververbose output. This mode is switched off by default.
HOST_COMPILER_FLAGS
- Flags that will be passed to the host compiler.
DEVICE_COMPILER_FLAGS
- Flags that will be passed to the device compiler.
Launching the test suite
The SYCL test suite can be launched via the following command:
$ python runtests.py --help usage: runtests.py [-h] [-b BINPATH] [--csvpath CSVPATH] [--list] [-j JUNIT] [--device DEVICE] Khronos SYCL CTS optional arguments: -h, --help show this help message and exit -b BINPATH, --binpath BINPATH specify path to the cts executable file --csvpath CSVPATH specify path to csv file for filtering tests --list list all tests in a test binary -j JUNIT, --junit JUNIT specify output path for a junit xml file -p PLATFORM, --platform PLATFORM The platform to run on (where PLATFORM can be host, intel, amd) -d DEVICE, --device DEVICE The device to run on (where DEVICE can be host, opencl_cpu, opencl_gpu, opencl_accelerator)
The --binpath
argument is mandatory and must point to one of the CTS
test executables built in the previous step.
An optional CSV file can be given which can be used to narrow the range of tests that will be executed.
The filters work using the principal of partial string matching. Any test in a CTS executable with a name that begins with one of the items in CSV file will be scheduled to be run. Those tests that don't match will not be run.
In the future the CSV file will also be used to specify timeout values on a per-test basis.
The --list
argument can be used to examine all of the tests that are
stored in a test executable. For instance:
$ python runtests.py -b build\bin\test_context.exe --list 3 tests in executable . context_api . context_constructors . context_getinfo
Passing the --junit
option will output test results in junit format
when the test suite has finished executing.
The --device
argument can be used to specify which device to run the
tests on. Selection is based on substring matching of the device name.
ECMAScript regular expression syntax is supported.
The following command will start a typical test run:
$ python runtests.py --binpath tests/common/test_all
During testing any fails will be reported with details about the failure. The following failure importantly shows the source file containing the test and the line number that signaled the failure:
platform_api: ? note: sycl exception caught ? note: what - Failed to get platform information. + result: fail ! file: ../../tests/platform/platform_api.cpp ! built: Aug 22 2017, 18:06:45 ! line: 96
After the test suite is finished a summary is produced helping programmers quickly identify failures and conformance rate:
16 tests ran in total - passed : 13 - failed : 1 + platform_api - skipped: 2 - 81% pass rate
This report should be packaged with the run tests and sent to Khronos for conformance submission.
Conformance submission
The conformance submission requires the use of the core.csv
filter alongside
with the run_conformance_tests.py
script. Please, note that this script will
switch the SYCL_CTS_ENABLE_FULL_CONFORMANCE
option on.