rohanpadhye/JQF

Connection refused error while trying to make a HTTP request in a test

enzocardeal opened this issue · 12 comments

enzocardeal commented

I'm trying to make a HTTP request in a test but am getting a Connection refused error by the end of it and no code coverage, so I assume not a single test was run.

With an identical test code, I am able to run some Junit tests and it works just fine.

For the HTTP server, I am using com.sun.net.httpserver.HttpServer and for making the request, I'm using java.net module.

I'm getting the following error:

[ERROR] Failed to execute goal edu.berkeley.cs.jqf:jqf-maven-plugin:1.8:fuzz (default-cli) on project back: Fuzzing resulted in the test failing on 466 input(s). Possible bugs found. Use mvn jqf:repro to reproduce failing test cases from /Projects/back/target/fuzz-results/br.usp.pcs.back.api.user.ApiFuzzTest/userSignUpTest/failures. Sample exception included with this message.: java.net.ConnectException: Connection refused -> [Help 1]

rohanpadhye commented

Hi. This looks like JQF is generating an input that lead to an ConnectException in your code. I can't say why that is as it is specific to your application. JQF thinks it has found a "bug" in your code, but perhaps it was just a driver configuration issue. Are any of the connection parameters coming from the fuzzed input (e.g., the hostname, port, or protocol)? As the message states, you can use mvn jqf:repro to run your test method with the saved file and hopefully you should see the ConnectException again with a stacktrace, and then you can debug it as you wish by adding logging statements or running in a debugger. See the wiki docs for how to run the repro command. Let me know if that helps.

enzocardeal commented

Hi. No connection parameters are comming from the fuzzed input. As test input, i'm using InputStream so I can use some controlled input as initial seed. The problem is that even with these controlled input that works on the Junit test, the test breaks with ConnectException and 0% of code covered.

I'm trying to better debug it with jqf:repro, but most part of the log is trimmed as follows:
There was 1 failure:

  1. userLoginTest(br.usp.pcs.back.api.user.ApiFuzzTest)
    java.lang.RuntimeException: java.net.ConnectException: Connection refused: connect
    at br.usp.pcs.back.api.user.ApiFuzzTest.postRequest(ApiFuzzTest.java)
    at br.usp.pcs.back.api.user.ApiFuzzTest.userLoginTest(ApiFuzzTest.java)
    ... 45 trimmed

I did set trimStackTrace false on the configuration of maven-surefire-plugin but the log is still trimmed. Is it a configuration of jqf plugin?

rohanpadhye commented

The log trimming is done by JUnit itself to avoid the noise of the boilerplate JUnit code that happens before your test method, so that's actually good. The point is that if you can reliably repro the failure then you can potentially debug it by adding print statements or attaching a breakpoint via a standard debugger. JQF-repro can help you reproduce a symptom, but debugging the root cause is up to you.

The JQF config cannot change anything specific to your application like network connections, though perhaps there is something that your test sets up beforehand that is not getting completed when running the JQF fuzzer?

One simple way to see the difference between JUnit and JQF is to create a dummy test method that takes exactly one input and ignores it, but does the same thing as the JUnit test.

@Test
public void testFoo() {
  /* some code */
}

@Fuzz
public void fuzzFoo(int _ignore_) {
  /* same code */
}

When you run mvn jqf:repro for fuzzFoo with ANY input file, you should get the same behavior as running the basic JUnit test testFoo. Can you try these both and tell me if you see a difference? If yes, I will need more info about the rest of the test to think of a reason why it differs (e.g., are there any @Before or @BeforeTest annotations or any other dependency injection frameworks being used, what kind of network connection you are attempting, etc.)

enzocardeal commented

Hi, I've managed to fix the issue.

The problem was I was using @BeforeAll annotation from JUnit5 in a method to instantiate the server. I saw in an example that you've used @Before from JUnit4, but it seems that with @BeforeAll or @BeforeEach, the method annotated is not executed.

Now I'm having another issue with mockito-all version 1.10.19. Without it, the test run fine. With it, running jqf:fuzz with -Djanala.verbose flag, I've got the following log:

[INFO] --- jqf-maven-plugin:1.8:fuzz (default-cli) @ app ---
[INFO] Instrumenting: br/usp/pcs/back/api/user/ApiFuzzTest... Done!
[INFO] Instrumenting: org/json/simple/parser/ParseException... Done!
[INFO] Instrumenting: org/json/simple/JSONObject... Done!
[INFO] Instrumenting: org/json/simple/JSONAware... Done!
[INFO] Instrumenting: org/json/simple/JSONStreamAware... Done!
[INFO] Instrumenting: br/usp/pcs/back/data/datasource/UserDataSource... Done!
[INFO] Instrumenting: org/mockito/Mockito... Done!
[INFO] Instrumenting: org/mockito/Matchers... Done!
[INFO] Instrumenting: org/mockito/internal/progress/MockingProgress... Done!
[INFO] Instrumenting: org/mockito/verification/VerificationMode... Done!
[INFO] Instrumenting: org/mockito/stubbing/Answer... Done!
[INFO] Instrumenting: org/mockito/verification/VerificationWithTimeout... Done!
[INFO] Instrumenting: org/mockito/verification/VerificationAfterDelay... Done!
[INFO] Instrumenting: org/mockito/MockitoDebugger... Done!
[INFO] Instrumenting: org/mockito/internal/progress/ThreadSafeMockingProgress... Done!
[INFO] Instrumenting: org/mockito/internal/MockitoCore... Done!
[INFO] Instrumenting: org/mockito/internal/verification/api/VerificationDataInOrder... Done!
[INFO] Instrumenting: org/mockito/exceptions/misusing/NotAMockException... Done!
[INFO] Instrumenting: org/mockito/exceptions/base/MockitoException... Done!
[INFO] Instrumenting: org/mockito/internal/verification/api/VerificationData... Done!
[INFO] Instrumenting: org/mockito/internal/stubbing/InvocationContainer... Done!
[INFO] Instrumenting: org/mockito/MockingDetails... Done!
[INFO] Instrumenting: org/mockito/InOrder... Done!
[INFO] Instrumenting: org/mockito/exceptions/Reporter... Done!
[INFO] Instrumenting: org/mockito/exceptions/misusing/NullInsteadOfMockException... Done!
[INFO] Instrumenting: org/mockito/exceptions/misusing/MissingMethodInvocationException... Done!
[INFO] Instrumenting: org/mockito/exceptions/verification/NeverWantedButInvoked... Done!
[INFO] Instrumenting: org/mockito/exceptions/base/MockitoAssertionError... Done!
[INFO] Instrumenting: org/mockito/exceptions/verification/TooManyActualInvocations... Done!
[INFO] Instrumenting: org/mockito/exceptions/verification/VerificationInOrderFailure... Done!
[INFO] Instrumenting: org/mockito/exceptions/verification/WantedButNotInvoked... Done!
[INFO] Instrumenting: org/mockito/exceptions/misusing/InvalidUseOfMatchersException... Done!
[INFO] Instrumenting: org/mockito/exceptions/misusing/CannotVerifyStubOnlyMock... Done!
[INFO] Instrumenting: org/mockito/exceptions/misusing/UnfinishedVerificationException... Done!
[INFO] Instrumenting: org/mockito/exceptions/misusing/UnfinishedStubbingException... Done!
[INFO] Instrumenting: org/mockito/exceptions/misusing/FriendlyReminderException... Done!
[INFO] Instrumenting: org/mockito/exceptions/verification/SmartNullPointerException... Done!
[INFO] Instrumenting: org/mockito/exceptions/misusing/WrongTypeOfReturnValue... Done!
[INFO] Instrumenting: org/mockito/exceptions/misusing/CannotStubVoidMethodWithReturnValue... Done!
[INFO] Instrumenting: org/mockito/exceptions/verification/NoInteractionsWanted... Done!
[INFO] Instrumenting: org/mockito/exceptions/verification/TooLittleActualInvocations... Done!
[INFO] Instrumenting: org/mockito/internal/util/MockUtil... Done!
[INFO] Instrumenting: org/mockito/invocation/MockHandler... Done!
[INFO] Instrumenting: org/mockito/mock/MockName... Done!
[INFO] Instrumenting: org/mockito/internal/configuration/plugins/Plugins... Done!
[INFO] Instrumenting: org/mockito/internal/configuration/plugins/PluginRegistry... Done!
[INFO] Instrumenting: org/mockito/plugins/PluginSwitch... Done!
[INFO] Instrumenting: org/mockito/internal/configuration/plugins/PluginLoader... Done!
[INFO] Instrumenting: org/mockito/exceptions/misusing/MockitoConfigurationException... Done!
[INFO] Instrumenting: org/mockito/internal/configuration/plugins/DefaultPluginSwitch... Done!
[INFO] Instrumenting: org/mockito/internal/configuration/plugins/PluginFinder... Done!
[INFO] Instrumenting: org/mockito/internal/util/collections/Iterables... Done!
[INFO] Instrumenting: org/mockito/plugins/MockMaker... Done!
[INFO] Instrumenting: org/mockito/internal/creation/cglib/CglibMockMaker... Done!
[INFO] Instrumenting: org/mockito/cglib/proxy/Callback... Done!
[INFO] Instrumenting: org/mockito/cglib/proxy/MethodInterceptor... Done!
[INFO] Instrumenting: org/mockito/plugins/StackTraceCleanerProvider... Done!
[INFO] Instrumenting: org/mockito/internal/exceptions/stacktrace/DefaultStackTraceCleanerProvider... Done!
[INFO] Instrumenting: org/mockito/Answers... Done!
[INFO] Instrumenting: org/mockito/internal/stubbing/defaultanswers/GloballyConfiguredAnswer... Done!
[INFO] Instrumenting: org/mockito/internal/stubbing/defaultanswers/ReturnsSmartNulls... Done!
[INFO] Instrumenting: org/mockito/invocation/Location... Done!
[INFO] Instrumenting: org/mockito/internal/stubbing/defaultanswers/ReturnsMoreEmptyValues... Done!
[INFO] Instrumenting: org/mockito/internal/stubbing/defaultanswers/ReturnsEmptyValues... Done!
[INFO] Instrumenting: org/mockito/internal/util/ObjectMethodsGuru... Done!
[INFO] Instrumenting: org/mockito/internal/invocation/MockitoMethod... Done!
[INFO] Instrumenting: org/mockito/internal/invocation/AbstractAwareMethod... Done!
[INFO] Instrumenting: org/mockito/internal/stubbing/defaultanswers/ReturnsMocks... Done!
[INFO] Instrumenting: org/mockito/internal/stubbing/defaultanswers/ReturnsDeepStubs... Done!
[INFO] Instrumenting: org/mockito/internal/stubbing/defaultanswers/ReturnsDeepStubs$ReturnsDeepStubsSerializationFallback... Done!
[INFO] Instrumenting: org/mockito/internal/stubbing/answers/CallsRealMethods... Done!
[INFO] Instrumenting: org/mockito/internal/creation/MockSettingsImpl... Done!
[INFO] Instrumenting: org/mockito/MockSettings... Done!
[INFO] Instrumenting: org/mockito/mock/MockCreationSettings... Done!
[INFO] Instrumenting: org/mockito/internal/creation/settings/CreationSettings... Done!
[INFO] Instrumenting: org/mockito/mock/SerializableMode... Done!
[INFO] Instrumenting: org/mockito/internal/util/MockCreationValidator... Done!
[INFO] Instrumenting: org/mockito/internal/util/MockNameImpl... Done!
[INFO] Instrumenting: org/mockito/internal/handler/MockHandlerFactory... Done!
[INFO] Instrumenting: org/mockito/internal/InternalMockHandler... Done!
[INFO] Instrumenting: org/mockito/internal/handler/MockHandlerImpl... Done!
[INFO] Instrumenting: org/mockito/stubbing/VoidMethodStubbable... Done!
[INFO] Instrumenting: org/mockito/internal/progress/IOngoingStubbing... Done!
[INFO] Instrumenting: org/mockito/invocation/InvocationOnMock... Done!
[INFO] Instrumenting: org/mockito/internal/invocation/MatchersBinder... Done!
[INFO] Instrumenting: org/mockito/internal/stubbing/InvocationContainerImpl... Done!
[INFO] Instrumenting: org/mockito/invocation/DescribedInvocation... Done!
[INFO] Instrumenting: org/mockito/exceptions/PrintableInvocation... Done!
[INFO] Instrumenting: org/mockito/invocation/StubInfo... Done!
[INFO] Instrumenting: org/mockito/internal/verification/SingleRegisteredInvocation... Done!
[INFO] Instrumenting: org/mockito/internal/verification/RegisteredInvocations... Done!
[INFO] Instrumenting: org/mockito/internal/verification/DefaultRegisteredInvocations... Done!
[INFO] Instrumenting: org/mockito/internal/util/collections/ListUtil$Filter... Done!
[INFO] Instrumenting: org/mockito/internal/handler/NullResultGuardian... Done!
[INFO] Instrumenting: org/mockito/internal/handler/InvocationNotifierHandler... Done!
[INFO] Instrumenting: org/mockito/listeners/MethodInvocationReport... Done!
[INFO] Instrumenting: org/mockito/internal/creation/cglib/AcrossJVMSerializationFeature... Done!
[INFO] Instrumenting: org/mockito/exceptions/base/MockitoSerializationIssue... Done!
[INFO] Instrumenting: org/mockito/internal/creation/cglib/ClassImposterizer... Done!
[INFO] Instrumenting: org/mockito/cglib/core/CodeGenerationException... Done!
[INFO] Instrumenting: org/mockito/cglib/proxy/Enhancer... Done!
[INFO] Instrumenting: org/mockito/cglib/core/AbstractClassGenerator... Done!
[INFO] Instrumenting: org/mockito/cglib/core/ClassGenerator... Done!
[INFO] Instrumenting: org/mockito/internal/creation/cglib/ClassImposterizer$3... Done!
[INFO] Instrumenting: org/mockito/cglib/core/NamingPolicy... Done!
[INFO] Instrumenting: org/mockito/internal/creation/instance/InstantationException... Done!
[INFO] Instrumenting: org/mockito/cglib/proxy/CallbackFilter... Done!
[INFO] Instrumenting: org/mockito/internal/creation/cglib/ClassImposterizer$1... Done!
[INFO] Instrumenting: org/mockito/internal/creation/cglib/MockitoNamingPolicy... Done!
[INFO] Instrumenting: org/mockito/cglib/core/DefaultNamingPolicy... Done!
[INFO] Instrumenting: org/mockito/internal/creation/cglib/ClassImposterizer$2... Done!
[INFO] Instrumenting: org/mockito/internal/creation/instance/InstantiatorProvider... Done!
[INFO] Instrumenting: org/mockito/internal/creation/instance/Instantiator... Done!
[INFO] Instrumenting: org/mockito/internal/creation/instance/ObjenesisInstantiator... Done!
[INFO] Instrumenting: org/objenesis/ObjenesisStd... Done!
[INFO] Instrumenting: org/objenesis/ObjenesisBase... Done!
[INFO] Instrumenting: org/objenesis/Objenesis... Done!
[INFO] Instrumenting: org/objenesis/strategy/InstantiatorStrategy... Done!
[INFO] Instrumenting: org/mockito/internal/configuration/GlobalConfiguration... Done!
[INFO] Instrumenting: org/mockito/configuration/IMockitoConfiguration... Done!
[INFO] Instrumenting: org/mockito/configuration/DefaultMockitoConfiguration... Done!
[INFO] Instrumenting: org/mockito/configuration/AnnotationEngine... Done!
[INFO] Instrumenting: org/mockito/internal/configuration/ClassPathLoader... Done!
[INFO] Instrumenting: org/objenesis/strategy/StdInstantiatorStrategy... Done!
[INFO] Instrumenting: org/objenesis/strategy/BaseInstantiatorStrategy... Done!
[INFO] Instrumenting: org/objenesis/instantiator/ObjectInstantiator... Done!
[INFO] Instrumenting: org/mockito/internal/creation/cglib/MethodInterceptorFilter... Done!
[INFO] Instrumenting: org/mockito/internal/creation/util/MockitoMethodProxy... Done!
[INFO] Instrumenting: org/mockito/internal/invocation/realmethod/RealMethod... Done!
[INFO] Instrumenting: org/mockito/invocation/Invocation... Done!
[INFO] Instrumenting: br/usp/pcs/back/data/repository/UserRepository... Done!
[INFO] Instrumenting: org/mockito/cglib/core/GeneratorStrategy... Done!
[INFO] Instrumenting: org/mockito/cglib/core/Predicate... Done!
[INFO] Instrumenting: org/mockito/cglib/proxy/CallbackGenerator$Context... Done!
[INFO] Instrumenting: org/mockito/cglib/core/ObjectSwitchCallback... Done!
[INFO] Instrumenting: org/mockito/cglib/core/ProcessSwitchCallback... Done!
[INFO] Instrumenting: org/mockito/cglib/core/Transformer... Done!
[INFO] Instrumenting: org/mockito/cglib/proxy/Enhancer$1... Done!
[INFO] Instrumenting: org/mockito/cglib/core/AbstractClassGenerator$Source... Done!
[INFO] Instrumenting: org/mockito/cglib/proxy/Enhancer$EnhancerKey... Done!
[INFO] Instrumenting: org/mockito/cglib/core/KeyFactory... Done!
[INFO] Instrumenting: org/mockito/cglib/core/Customizer... Done!
[INFO] Instrumenting: org/mockito/cglib/core/TypeUtils... Done!
[INFO] Instrumenting: org/mockito/cglib/core/CollectionUtils... Done!
[INFO] Instrumenting: org/mockito/cglib/core/Signature... Done!
[INFO] Instrumenting: org/mockito/asm/Type... Done!
[INFO] Instrumenting: org/mockito/cglib/core/KeyFactory$1... Done!
[INFO] Instrumenting: org/mockito/cglib/core/KeyFactory$2... Done!
[INFO] Instrumenting: org/mockito/cglib/core/KeyFactory$Generator... Done!
[INFO] Instrumenting: org/mockito/cglib/core/DefaultGeneratorStrategy... Done!
[INFO] Instrumenting: org/mockito/asm/ClassWriter... Done!
[INFO] Instrumenting: org/mockito/asm/ClassVisitor... Done!
[INFO] Instrumenting: org/mockito/cglib/core/DebuggingClassWriter... Done!
[INFO] Instrumenting: org/mockito/asm/AnnotationVisitor... Done!
[INFO] Instrumenting: org/mockito/asm/FieldVisitor... Done!
[INFO] Instrumenting: org/mockito/asm/MethodVisitor... Done!
[INFO] Instrumenting: org/mockito/asm/ByteVector... Done!
[INFO] Instrumenting: org/mockito/asm/Item... Done!
[INFO] Instrumenting: org/mockito/cglib/core/ClassEmitter... Done!
[INFO] Instrumenting: org/mockito/asm/ClassAdapter... Done!
[INFO] Instrumenting: org/mockito/cglib/core/CodeEmitter... Done!
[INFO] Instrumenting: org/mockito/cglib/core/LocalVariablesSorter... Done!
[INFO] Instrumenting: org/mockito/asm/MethodAdapter... Done!
[INFO] Instrumenting: org/mockito/cglib/core/ClassEmitter$3... Done!
[INFO] Instrumenting: org/mockito/cglib/core/ClassInfo... Done!
[INFO] Instrumenting: org/mockito/cglib/core/ClassEmitter$1... Done!
[INFO] Instrumenting: org/mockito/cglib/core/ReflectUtils... Done!
[INFO] Instrumenting: org/mockito/cglib/core/MethodInfo... Done!
[INFO] Instrumenting: org/mockito/cglib/core/ReflectUtils$3... Done!
[INFO] Instrumenting: org/mockito/cglib/core/ReflectUtils$4... Done!
[INFO] Instrumenting: org/mockito/cglib/core/ReflectUtils$1... Done!
[INFO] Instrumenting: org/mockito/cglib/core/ReflectUtils$2... Done!
Fuzzing stopped due to guidance exception: java.lang.IllegalArgumentException: Input is either empty or nothing was requested from the input generator.
rohanpadhye commented

Thanks for the update. I'll look into why @BeforeAll is not getting executed with JQF.

For the latest issue, it seems that there is some test method with @Fuzz annotation that doesn't have formal parameters (e.g., a regular JUnit test) or perhaps an InputStream or similar parameter with lazy semantics where no bytes were ever read by the test method (e.g. test(InputStream foo) { }. Essentially, the fuzzer has no inputs to mutate.

enzocardeal commented

What I have is this:

    public void beforeEach(){

        try{
            userDataSourceMock = mock(UserDataSource.class);

            when(userDataSourceMock.get(eq("user1"))).thenReturn(new UserEntity(UUID.randomUUID(), "user1", hashedPassword, Role.DEFAULT));
            when(userDataSourceMock.get(eq("user2"))).thenReturn(new UserEntity(UUID.randomUUID(), "user2", hashedPassword, Role.DEFAULT));
            when(userDataSourceMock.get(eq("user3"))).thenReturn(new UserEntity(UUID.randomUUID(), "user3", hashedPassword, Role.DEFAULT));

            when(userDataSourceMock.get(eq("user4"))).thenReturn(null);

            when(userDataSourceMock.get(eq("user5"))).thenReturn(new UserEntity(UUID.randomUUID(), "user5", hashedPassword, Role.ADMIN));

            when(userDataSourceMock.create(eq("user1"), any())).thenReturn(new UserEntity(UUID.randomUUID(), "user1", hashedPassword, Role.DEFAULT));
            when(userDataSourceMock.create(eq("user2"), any())).thenReturn(new UserEntity(UUID.randomUUID(), "user2", hashedPassword, Role.DEFAULT));
            when(userDataSourceMock.create(eq("user3"), any())).thenReturn(new UserEntity(UUID.randomUUID(), "user3", hashedPassword, Role.DEFAULT));

            when(userDataSourceMock.create(eq("user4"), any())).thenReturn(null);
            when(userDataSourceMock.create(eq("user5"), any())).thenReturn(null);
            when(userDataSourceMock.create(eq("user6"), any())).thenReturn(null);

            LoginController loginController = new LoginController(userDataSourceMock, getObjectMapper(),
                    getErrorHandler());
            SignUpController signUpController = new SignUpController(userDataSourceMock, getObjectMapper(),
                    getErrorHandler());

            server = HttpServer.create(new InetSocketAddress(serverPort), 0);
            server.createContext("/api/user/login", loginController::handle);
            server.createContext("/api/user/signup", signUpController::handle);
            server.setExecutor(null);
            server.start();
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
    }

    @Fuzz
    public void userLoginTest(InputStream userInput){
        beforeEach();
        List<String> userInputSplit = parseUserInput(userInput);
        String username = userInputSplit.get(0);
        String password = userInputSplit.get(1);
        postRequest("/api/user/login", username, password);
    }

If I comment all mockito related code like this:

    public void beforeEach(){

        try{
//            userDataSourceMock = mock(UserDataSource.class);
//
//            when(userDataSourceMock.get(eq("user1"))).thenReturn(new UserEntity(UUID.randomUUID(), "user1", hashedPassword, Role.DEFAULT));
//            when(userDataSourceMock.get(eq("user2"))).thenReturn(new UserEntity(UUID.randomUUID(), "user2", hashedPassword, Role.DEFAULT));
//            when(userDataSourceMock.get(eq("user3"))).thenReturn(new UserEntity(UUID.randomUUID(), "user3", hashedPassword, Role.DEFAULT));
//
//            when(userDataSourceMock.get(eq("user4"))).thenReturn(null);
//
//            when(userDataSourceMock.get(eq("user5"))).thenReturn(new UserEntity(UUID.randomUUID(), "user5", hashedPassword, Role.ADMIN));
//
//            when(userDataSourceMock.create(eq("user1"), any())).thenReturn(new UserEntity(UUID.randomUUID(), "user1", hashedPassword, Role.DEFAULT));
//            when(userDataSourceMock.create(eq("user2"), any())).thenReturn(new UserEntity(UUID.randomUUID(), "user2", hashedPassword, Role.DEFAULT));
//            when(userDataSourceMock.create(eq("user3"), any())).thenReturn(new UserEntity(UUID.randomUUID(), "user3", hashedPassword, Role.DEFAULT));
//
//            when(userDataSourceMock.create(eq("user4"), any())).thenReturn(null);
//            when(userDataSourceMock.create(eq("user5"), any())).thenReturn(null);
//            when(userDataSourceMock.create(eq("user6"), any())).thenReturn(null);

            LoginController loginController = new LoginController(userDataSourceMock, getObjectMapper(),
                    getErrorHandler());
            SignUpController signUpController = new SignUpController(userDataSourceMock, getObjectMapper(),
                    getErrorHandler());

            server = HttpServer.create(new InetSocketAddress(serverPort), 0);
            server.createContext("/api/user/login", loginController::handle);
            server.createContext("/api/user/signup", signUpController::handle);
            server.setExecutor(null);
            server.start();
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
    }

    @Fuzz
    public void userLoginTest(InputStream userInput){
        beforeEach();
        List<String> userInputSplit = parseUserInput(userInput);
        String username = userInputSplit.get(0);
        String password = userInputSplit.get(1);
        postRequest("/api/user/login", username, password);
    }

The test run just fine.

rohanpadhye commented

Thanks for the snippet! What does parseUserInput do? My first guess is whatever Mockito is doing could be preventing userInput.read() from being invoked, thus JQF thinks that your test has no input data.

enzocardeal commented

Thanks for the response!

This is the code for parseUserInput:

    public List<String> parseUserInput(InputStream inputStream){
        String userInputString = null;
        String user = null;
        String password = null;

        try {
            userInputString = convertInputStreamToString(inputStream);
        } catch (IOException e) {
            e.printStackTrace();
        }

        boolean hasComma = userInputString.contains(",");

        if(hasComma) {
            String[] userInputSplit = userInputString.split(",");
            user = userInputSplit[0];
            password = userInputSplit[1];
        }
        else {
            user = userInputString.substring(0, (userInputString.length()/2));
            password = userInputString.substring((userInputString.length()/2));
        }
        return Arrays.asList(user, password);
    }

And the code for convertInputStreamToString:

    public static final int DEFAULT_BUFFER_SIZE = 1024;

    public static String convertInputStreamToString(InputStream inputStream) throws IOException {

        ByteArrayOutputStream result = new ByteArrayOutputStream();
        byte[] buffer = new byte[DEFAULT_BUFFER_SIZE];
        int length;
        while ((length = inputStream.read(buffer)) != -1) {
            result.write(buffer, 0, length);
        }


        return result.toString("UTF-8");

    }

For the seed file, I'm having a string inside it as: "<username>,<password>".

As for the possible userInput.read() issue with Mockito, I'll use a dummy db instead and will get back with updates.

enzocardeal commented

Hi.

No success fuzzing with a running db server. I'm able to use it on a JUnit test, but with the fuzzer I get the following error:

java.lang.RuntimeException: java.sql.SQLException: No suitable driver found for jdbc:postgresql://localhost:5432/postgres?stringtype=unspecified
        at br.usp.pcs.back.data.dbconn.DbConn.connect(DbConn.java:17)
        at br.usp.pcs.back.data.repository.UserRepository.get(UserRepository.java:43)
        at br.usp.pcs.back.data.datasource.UserDataSource.get(UserDataSource.java:18)
        at br.usp.pcs.back.api.user.LoginController.doPost(LoginController.java:56)
        at br.usp.pcs.back.api.user.LoginController.execute(LoginController.java:37)
        at br.usp.pcs.back.api.Controller.lambda$0(Controller.java:24)
        at io.vavr.control.Try.run(Try.java:118)
        at br.usp.pcs.back.api.Controller.handle(Controller.java:24)
        at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:95)
        at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82)
        at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:98)
        at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:851)
        at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:95)
        at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:818)
        at jdk.httpserver/sun.net.httpserver.ServerImpl$DefaultExecutor.execute(ServerImpl.java:201)
        at jdk.httpserver/sun.net.httpserver.ServerImpl$Dispatcher.handle(ServerImpl.java:561)
        at jdk.httpserver/sun.net.httpserver.ServerImpl$Dispatcher.run(ServerImpl.java:527)
        at java.base/java.lang.Thread.run(Thread.java:833)

About Mockito, If I change the fuzzing method input for (String, String), it works just fine. The downside is that I don't have known good inputs as seeds. I think I'll try to build a generator as a work around.

rohanpadhye commented

Thanks for the updates! Do you know if your test drivers work with plain junit-quickcheck (which supports random parameterized tests but not coverage instrumentation)? Try it by changing @Fuzz to @Property and then just run the tests using mvn test as usual. Quickcheck should just provide 50-100 random samples and run your test. If that works but @Fuzz and/or mvn jqf:fuzz does not, then it may be a specific instrumentation/class-loading bug that I will have to track down.

enzocardeal commented

Hi,

I did manage to run it with @Proterty and with that, ended up managing to solve the problem with the @Fuzz annotated method. Instead of calling the beforeEach() method inside the fuzzing method, I've annotated it with @BeforeClass and with that, the problems with Mockito were solved.

The code is like this now:

    private static UserDataSource userDataSourceMock;
    private static String hashedPassword = "$argon2id$v=19$m=1048576,t=4,p=8$mKSTdK1csbCCRgmiCh/9lw$SsOq1K8++MDziplzjnFN5HwXarntBMFwsKz7eNQVKnQ";
    private static HttpServer server = null;
    private static int serverPort = 8000;
    private static String baseUrl = "http://localhost";

    @BeforeClass
    public static void beforeAll(){

        try{
            userDataSourceMock = mock(UserDataSource.class);

            when(userDataSourceMock.get(eq("user1"))).thenReturn(new UserEntity(UUID.randomUUID(), "user1", hashedPassword, Role.DEFAULT));
            when(userDataSourceMock.get(eq("user2"))).thenReturn(new UserEntity(UUID.randomUUID(), "user2", hashedPassword, Role.DEFAULT));
            when(userDataSourceMock.get(eq("user3"))).thenReturn(new UserEntity(UUID.randomUUID(), "user3", hashedPassword, Role.DEFAULT));

            when(userDataSourceMock.get(eq("user4"))).thenReturn(null);

            when(userDataSourceMock.get(eq("user5"))).thenReturn(new UserEntity(UUID.randomUUID(), "user5", hashedPassword, Role.ADMIN));

            when(userDataSourceMock.create(eq("user1"), any())).thenReturn(new UserEntity(UUID.randomUUID(), "user1", hashedPassword, Role.DEFAULT));
            when(userDataSourceMock.create(eq("user2"), any())).thenReturn(new UserEntity(UUID.randomUUID(), "user2", hashedPassword, Role.DEFAULT));
            when(userDataSourceMock.create(eq("user3"), any())).thenReturn(new UserEntity(UUID.randomUUID(), "user3", hashedPassword, Role.DEFAULT));

            when(userDataSourceMock.create(eq("user4"), any())).thenReturn(null);
            when(userDataSourceMock.create(eq("user5"), any())).thenReturn(null);
            when(userDataSourceMock.create(eq("user6"), any())).thenReturn(null);

            LoginController loginController = new LoginController(userDataSourceMock, getObjectMapper(),
                    getErrorHandler());
            SignUpController signUpController = new SignUpController(userDataSourceMock, getObjectMapper(),
                    getErrorHandler());

            server = HttpServer.create(new InetSocketAddress(serverPort), 0);
            server.createContext("/api/user/login", loginController::handle);
            server.createContext("/api/user/signup", signUpController::handle);
            server.setExecutor(null);
            server.start();
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
    }

    @Fuzz
    public void userLoginTest(InputStream userInput){
        List<String> userInputSplit = parseUserInput(userInput);
        String username = userInputSplit.get(0);
        String password = userInputSplit.get(1);
        postRequest("/api/user/login", username, password);
    }

Thank you for your time helping me solving this issue.

rohanpadhye commented

Ahh, I see. It seems that @BeforeEach and @BeforeAll are JUnit5 annotations, where in JUnit 4 (which is what JQF and Junit-Quickcheck support) they are called @Before and @BeforeClass respectively. I think using either of the latter two (depending on whether you want it to execute before every test instance execution or just once in the whole session) should ensure that the JQF runner executes the annotated methods properly.

Thanks for your patience and for reporting back!