[BUG] JQF can't catch failure when `throws Exception` is explicitly defined in under-fuzz method signature

[shuaiwang516]

Version

JQF 1.8, Java 11, Maven 3.6.3

Description

I found that JQF can't catch exceptions when the under-fuzz method has explicit throws Exception in the method signature.

Bug Reproduction

When fuzzing the following method testFakeFailure1(), JQF can successfully catch failure and record it.

  @Fuzz
  public void testFakeFailure1(String str) {
    throw new RuntimeException("Fake Failure1 " + str);
  }

However, when adding throws Exception in the method signature, JQF can't catch the "fake bug" and return success.

  @Fuzz
  public void testFakeFailure2(String str) throws Exception {   // <---- adding `throws Exception` in the method signature
    throw new RuntimeException("Fake Failure2 " + str);
  }

[rohanpadhye]

Thanks for the report! This is by design. The throws clause can be used to specify expected exceptions that don't indicate failure. The wiki contains some examples: https://github.com/rohanpadhye/JQF/wiki/Writing-a-JQF-test#jdk-datetimeformatter. This is a handy way to separate benign exceptions from unexpected failures.

I would recommend not annotating test methods with very broad throws declarations like Exception as that will swallow most failures.

[shuaiwang516]

Thank you for the explanation! I understand the design purpose now :)

[rohanpadhye]

For reference, the logic implementing this feature is in FuzzStatement.java. I am happy to make this configurable by adding a system property that can enable/disable this feature via the command-line.

[shuaiwang516]

I've created a PR to add this feature: #197.

Thanks again for your time!