yarpgen is a random program generator, which produces correct runnable C/C++ and DPC++(this work is in an early stage) programs. The generator is specifically designed to trigger compiler optimization bugs and is intended for compiler testing.
A generated random program is guaranteed to be statically and dynamically correct program. This implies no undefined behavior, but allows for implementation defined behavior.
Currently, there exist two versions of YARPGen: one is designed to test loops (it is under development and lives in the main branch), and the other one is designed to test scalar optimizations (it lives in v1 branch). The information about YARPGen and loop version can be found in this talk and this paper, published at PLDI23. More information about scalar version can be found in this talk and this paper, published at the OOPSLA 2020 and received an ACM SIGPLAN distinguished paper award.
Each generated program consists of several files and after being compiled and run produces a decimal number, which is hash of all program global variable values. This number is supposed to be the same for all compilers and optimization levels. If the output differs for different compilers and/or optimization levels, you should have encountered a compiler bug.
Several techniques are used to increase the probability of triggering compiler bugs:
- all variable types, alignments and values / value ranges are known at the time of generation, which allows the accurate detection of undefined behavior and provides maximum variability of produced code.
- the generation is random, but it is guided by a number of policies, which increase the likelihood of applying optimizations to the generated code.
- in some cases a high level model of computations with known properties is generated first and then augmented with other random computations. This ensures that the code is "meaningful" and is more similar to code written by a human being.
yarpgen managed to find more than 260 bugs in gcc, clang, ispc, dpc++, sde, alive2 and a comparable number of bugs in
proprietary compilers. For the list of the bugs in gcc, clang, ispc, and alive2 see bugs.rst.
If you have used yarpgen to find bugs in open source compilers, please update bugs.rst. We are also excited to hear about your experience using it with proprietary compilers and other tools you might want to validate.
Building yarpgen is trivial. All you have to do is to use cmake:
mkdir build
cd build
cmake ..
makeTo run yarpgen we recommend using run_gen.py script, which will run the generator for you
on a number of available compilers with a set of pre-defined options. Feel free to hack
test_sets.txt to add or remove compiler options.
The script will run several compilers with several compiler options and run executables to compare the output results. If the results mismatch, the test program will be saved in "testing/results" folder for your analysis.
Also you may want to test compilers for future hardware, which is not available to you at the moment. The standard way to do that is to download the Intel® Software Development Emulator. run_gen.py assumes that it is available in your $PATH.
If you want to test ISPC, make sure that it is present in your path, as well as
ispc-proxy and ispc-disp. After that you can use
run_gen.py with --std=ispc as usual.
To contact authors, ask questions, or leave your feedback please use Github issues or reach out directly through contacts available in Github profiles.
- Vsevolod Livinskii
- Dmitry Babokin
- John Regehr
Special thanks to Martin Liška and Detlef Riekenberg who used YARPGen to find bugs in various compilers.