This crate implements the DudeCT statistical methods for
testing constant-time functions. It is based loosely off of the
bencher crate.
Example use is as follows. Since this requires the current crate as a dependency, it is easiest to
put the benchmarks in examples/. Take a look at examples/ctbench-foo.rs for sample source code.
To run all the benchmarks in examples/ctbench-foo.rs, you can simply run cargo run --release --example ctbench-foo.
To run a subset of the benchmarks in the above file that have a the substring ar in it, run
cargo run --release --example ctbench-foo -- --filter ar.
To run the vec_eq benchmark continuously, collecting more samples as it goes along, run cargo run --release --example ctbench-foo -- --continuous vec_eq.
To run the benchmarks in ctbench-foo and get the raw runtimes in CSV format, run cargo run --release --example ctbench-foo -- --out data.csv.
The benchmark output looks like
bench array_eq ... : n == +0.046M, max t = +61.61472, max tau = +0.28863, (5/tau)^2 = 300
It is interpreted as follows. Firstly note that the runtime distributions are cropped at different
percentiles and about 100 t-tests are performed. Of these t-tests, the one that produces the largest
absolute t-value is printed as max_t. The other values printed are
n, indicating the number of samples used in computing this t-valuemax_tau, which is the t-value scaled for the samples size (formally,max_tau = max_t / sqrt(n))(5/tau)^2, which indicates the number of measurements that would be needed to distinguish the two distributions with t > 5
t-values greater than 5 are generally considered a good indication that the function is not constant time. t-values less than 5 does not necessarily imply that the function is constant-time, since there may be other input distributions under which the function behaves significantly differently.
Again, this project derives from the bencher crate under the
MIT license. This project is licensed under the (MIT license) as well.