Csmith is a random generator of C programs. It's primary purpose is to find compiler bugs with random programs, using differential testing as the test oracle.
Csmith can be used outside of the field of compiler testing. If your application needs a test suite of C programs and you don't bother to write them, feel free to give Csmith a try.
Csmith outputs C programs free of undefined behaviors (believe us, that's not trivial), and the statistics of each generated program.
You can install Csmith from tarballs downloaded from here (coming soon), or you can build it from the source. The following commands apply to Ubuntu.
git clone https://github.com/csmith-project/csmith.git
cd csmith
sudo apt install g++ cmake m4
cmake -DCMAKE_INSTALL_PREFIX=<INSTALL-PREFIX> .
make && make install
Please see specific instructions for building on Windows.
Suppose Csmith is installed to $HOME/csmith locally. You can simply
generate, compile, and execute a test case by:
export PATH=$PATH:$HOME/csmith/bin
csmith > random1.c
gcc random1.c -I$HOME/csmith/include -o random1
./random1To add differential testing into the picture, we need to install another compiler, e.g., another version of gcc or clang. And repeat the process of:
csmith > random2.c
gcc random2.c -I$HOME/csmith/include -o random2_gcc
clang random2.c -I$HOME/csmith/include -o random2_clang
./random2_gcc > gcc_output.txt
./random2_clang > clang_output.txt
If there is any difference in gcc_output.txt and clang_output.txt,
aha, you have found a bug in either gcc or clang, or, in the
unlikely case, a bug in Csmith itself.
You could write scripts in your favorite language to repeat the above process to amplify the power of random differential testing.
The generate programs might contain infinite loops. The best practice is to apply timeout to their executions.
Use csmith -h or csmith -hh to see lists of command line options that you
can pass to Csmith and customize the random generation.
Here is a slightly outdated but still relevant document about using Csmith for compiler testing.
Csmith was originally developed at the University of Utah by:
as part of a research project on compiler testing. The research is best summarized by our paper Finding and Understanding Bugs in C Compilers. More research info can be found here.
Csmith was open sourced in 2009. We try to keep maintaining it as an open source project using discretionary times. As much, the response to bug reports or feature requests might be delayed.
Please use github issues to report bugs or suggestions.
We have a mailing list for discussing Csmith. Please visit here to subscribe.