Potential issue in Zest/tutorial 1

[YuqiHuai]

Hi! I am writing about a potential issue with Zest tutorial 1. The issue is reproduced both on Ubuntu and Mac OS. I am following all the steps until Optional: Test with JUnit and QuickCheck

My folder structure is

JQF
/ bin
/ examples
/ fuzz
/ instrument
/ tutorial 
  / CalendarLogic.java
  / CalendarGenerator.java
  / CalendarTest.java
...

To Reproduce

In the directory tutorial, I run

java -cp .:$(../scripts/classpath.sh) org.junit.runner.JUnitCore CalendarTest

Expecting

Assumption is too strong; too many inputs discarded

Actual

java.lang.NoClassDefFoundError: edu/berkeley/cs/jqf/instrument/InstrumentationException
        at edu.berkeley.cs.jqf.fuzz.JQF.methodBlock(JQF.java:187)
        at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
        at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
......

To fix

I do not know Java well but I tried to fix the issue. I added the following lines to JQF/scripts/classpath.sh after line 17, adding classes/jars from instrument to $cp.

cp="$cp:$ROOT_DIR/instrument/target/classes:$ROOT_DIR/instrument/target/test-classes"
for jar in $ROOT_DIR/instrument/target/dependency/*.jar; do
  cp="$cp:$jar"
done

and now I am getting the expected output

1) testLeapYear(CalendarTest)
edu.berkeley.cs.jqf.fuzz.guidance.GuidanceException: Assumption is too strong; too many inputs discarded
        at edu.berkeley.cs.jqf.fuzz.random.NoGuidance.handleResult(NoGuidance.java:124)
        at edu.berkeley.cs.jqf.fuzz.junit.quickcheck.FuzzStatement.evaluate(FuzzStatement.java:176)
        at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)

Question

Should the final variable $cp from scripts/classpath.sh include instrument/target/classes, instrument/target/test-classes and every jar in instrument/target/dependency (actually just 1 jar)?

[rohanpadhye]

Hi @YuqiHuai. Thanks for pointing this out! I was able to repro the issue on my end as well. I don't know when the tutorial instructions broke, perhaps with the last change to fuzz/pom.xml or the classpath.sh file itself. But, you are right, that currently the script does not include classes from the instrument package in the classpath, but it should! The four lines that you suggested are correct. I'll push a quick fix.

[YuqiHuai]

@rohanpadhye sounds good! I'll continue to explore! I have a follow-up question that might not be an issue. So I am still in the same tutorial and I am running ../bin/jqf-zest -c .:$(../scripts/classpath.sh) CalendarTest testLeapYear

I am expecting something like what the tutorial is suggesting

java.lang.AssertionError: 3600 should be a leap year
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at CalendarTest.testLeapYear(CalendarTest.java:21)

But instead, I just keep getting too many trials without coverage

Semantic Fuzzing with Zest
--------------------------

Test name:            CalendarTest#testCompare
Instrumentation:      Janala
Semantic Fuzzing with Zest
--------------------------

Test name:            CalendarTest#testLeapYear
Instrumentation:      Janala
Results directory:    /path/to/JQF/tutorial/fuzz-results
Elapsed time:         4s (no time limit)
Number of executions: 86,269 (no trial limit)
Valid inputs:         56 (0.06%)
Cycles completed:     0
Unique failures:      1
Queue size:           0 (0 favored last cycle)
Current parent input: <seed>
Execution speed:      41,603/sec now | 21,486/sec overall
Total coverage:       0 branches (0.00% of map)
Valid coverage:       0 branches (0.00% of map)
Fuzzing stopped due to guidance exception: Too many trials without coverage; likely all assumption violations
E
Time: 3.695
There was 1 failure:
1) testLeapYear(CalendarTest)
edu.berkeley.cs.jqf.fuzz.guidance.GuidanceException: Too many trials without coverage; likely all assumption violations
        at edu.berkeley.cs.jqf.fuzz.ei.ZestGuidance.lambda$getInput$0(ZestGuidance.java:672)
        at edu.berkeley.cs.jqf.fuzz.ei.ZestGuidance.conditionallySynchronize(ZestGuidance.java:1063)

Do you know if I am doing something incorrectly or is this also broken due to some updates?

[rohanpadhye]

Thanks again for pointing this out! It turns out the version of ASM in jqf-driver.sh was out of date, and needed updating. I just pushed another quick fix, and you should now see coverage when you run bin/jqf-zest.

Sorry about these issues! Most users directly run the Maven plugin via mvn jqf:fuzz, so the command-line scripts (the .sh files) tend to go out of date, breaking the tutorials.

[YuqiHuai]

@rohanpadhye Thank you so much for your timely replies and fixes! No worries! I am just getting started and have been following the tutorial. I haven't set up a maven project to run mvn jqf:fuzz but I will give it a try soon!