MozillaSecurity/lithium

Add ability to always treat the test case as interesting initially

dkasak opened this issue · 1 comments

The ability to specify that lithium should treat the test case as interesting initially (before any chunk removals are done, regardless of whether it is actually interesting or not) would be a nice addition.

It would allow lithium to act as a fuzzer, as explained in the Reducers are Fuzzers blog post.

Some use cases:

  • It often happens that I have a hunch that a certain part of the code is broken, but I'm unable to trigger the faulty behaviour with a test case. What sometimes works is writing a test case that exercises this part of the code but is uninteresting (e.g. doesn't crash) and then letting lithium crashes run while forcing it to treat the case as interesting.
  • A fix for a bug had been deployed but you have a hunch that the fix isn't a proper one and just fixes it for some particular test case. This would allow you to uncover potential related test cases that still trigger the bug.

I'm not sure how to implement this best in lithium, though. Any suggestions? For now, I've just hacked together a non-crashes interestingness test for my personal use. It behaves exactly as crashes, except it forces the test case to be interesting in the initial run. The downside of this approach is that you can't reuse this code with other interestingness tests for which you might want to turn this behaviour on.

Having a non-crashes interestingness test would probably be the best way forward.

Note that it is possible to hook in multiple interestingness tests together. e.g. it is possible to use the range interestingness test with the crashes one.

Sample portion of a CLI command:

range 1 20 crashes --timeout=3

will run each iteration 20 times before determining interestingness using the crashes test.