Benchmarks and non regression tests
Opened this issue · 7 comments
is there an easy way to add benchmarks to the non-regression tests. So that when we launch build.sh it does not end with "100% tests passed" but having the benchmarks' compilation which fails
We could add the benchmarks themselves as tests
Any progress on this?
The benchmarks take too much time for testing purposes. We can build them, but not automatically run them from the script, does that seems fine?
I do not remember it was so slow.
I have the feeling that currently the build script runs them anyway.
It is just that the results is not taken into account in the test success rate.
So it should not change the global run time, if my understanding is correct.
The CMakeList script currently builds the benchmarks but does not execute them. I am reluctant to add it to the testing, since it will probably bee too long for travis. Would adding a bash script to execute it be enough?
I have the feeling it is not explained in the documentation (the top-level README?) how to actually run the benchmarks. Only the compilation process is explained.