ClassNotFoundException raised when a user-defined class tries to load itself using ContextClassLoader during Fuzzing

[jiahaow4]

Problem

While trying to use JQF on a real-world project, I ran into another JQF limitation when the test code tries to load a class using ContextClassLoader. When jqf:fuzz is used, the test raises java.lang.ClassNotFoundException. When surefire:test is used, the test runs successfully.

Reproduce

1. Create a Maven project, add the following code in src/test/java/ExampleTest.java

import edu.berkeley.cs.jqf.fuzz.Fuzz;
import edu.berkeley.cs.jqf.fuzz.JQF;
import org.junit.runner.RunWith;
import org.junit.Test;

@RunWith(JQF.class)
public class ExampleTest {
    private void m() throws ClassNotFoundException {
        Class.forName(ExampleTest.class.getName(), true, Thread.currentThread().getContextClassLoader());
    }

    @Fuzz
    public void testFuzz(String str) throws Exception {
        m();
    }

    @Test
    public void test() throws Exception {
        m();
    }
}

2. Create the following pom.xml file. The JQF version I used is 2.0-SNAPSHOT.

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>example</groupId>
  <artifactId>example</artifactId>
  <version>1.0-SNAPSHOT</version>

  <name>example</name>

  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.13.2</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>edu.berkeley.cs.jqf</groupId>
      <artifactId>jqf-fuzz</artifactId>
      <version>2.0-SNAPSHOT</version>
    </dependency>
  </dependencies>

  <build>
    <pluginManagement>
      <plugins>
        <plugin>
          <artifactId>maven-clean-plugin</artifactId>
        </plugin>
        <plugin>
          <artifactId>maven-resources-plugin</artifactId>
        </plugin>
        <plugin>
          <artifactId>maven-compiler-plugin</artifactId>
        </plugin>
        <plugin>
          <artifactId>maven-surefire-plugin</artifactId>
        </plugin>
        <plugin>
          <artifactId>maven-jar-plugin</artifactId>
        </plugin>
        <plugin>
          <artifactId>maven-install-plugin</artifactId>
        </plugin>
        <plugin>
          <artifactId>maven-deploy-plugin</artifactId>
        </plugin>
        <plugin>
          <artifactId>maven-site-plugin</artifactId>
        </plugin>
        <plugin>
          <artifactId>maven-project-info-reports-plugin</artifactId>
        </plugin>
        <plugin>
          <groupId>edu.berkeley.cs.jqf</groupId>
          <artifactId>jqf-maven-plugin</artifactId>
          <version>2.0-SNAPSHOT</version>
        </plugin>
      </plugins>
    </pluginManagement>
  </build>
</project>

3. In the project's root directory, run mvn test. Then to see the difference, run mvn jqf:fuzz -Dclass=ExampleTest -Dmethod=testFuzz -Djqf.failOnDeclaredExceptions -DexitOnCrash -X for jqf:fuzz, and run mvn surefire:test -Dtest=ExampleTest#test for surefire:test.

Result

The following is the partial stack trace from jqf:fuzz. In surefire:test, the test ran successfully.

There was 1 failure:
1) testFuzz(ExampleTest)
java.lang.ClassNotFoundException: ExampleTest
	at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass(SelfFirstStrategy.java:50)
	at org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass(ClassRealm.java:271)
	at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:247)
	at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:239)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:398)
	at ExampleTest.m(ExampleTest.java:9)
	at ExampleTest.testFuzz(ExampleTest.java:14)

[rohanpadhye]

Thanks for the detailed issue report with repro! This looks like a bug, but I need to wrap my head around what it means for a class to load itself via the context-class-loader. I'll debug more and report back on what the issue is.

[rohanpadhye]

Alright, I found the issue. tl;dr---Just needed to reset the context class loader to be the same as the test application's class loader.

When running with JQF's Maven Plugin, the application classes are loaded with a custom classloader that has the test classes on the classpath. However, the thread's context class loader is still the parent loader that Maven uses to load itself (i.e., it only has Apache Maven, JQF, and their dependencies on the classpath). So, applications that use the thread's context class loader to load app classes are not able to find those classes.

Surefire (mvn test) doesn't seem to have this problem, because it launches new JVMs with the test application on the system class loader's classpath (and also the main thread's context loader).

Generally speaking, applications should load other app classes using their own class loader (i.e., getClass().getClassLoader()), but I'm aware that some frameworks use context hackery to get around name binding and such.

Anyway, the fix for this problem is simple and doesn't seem to break anything from what I can tell. Thanks again for the report. Let me know if you spot any other issues after updating.