Failing to reproduce bugs locally
Closed this issue · 5 comments
Hi,
I'm trying to reproduce some of GHRB's bugs locally, but for some reason failing.
I am running a slightly modified version (https://github.com/ASSERT-KTH/GHRB) since I need to have access to the sample files in my local filesystem.
The only difference is that I bind a temporary dir to the container that is launched.
Note that these do not always occur, I've tried in different setups with different results.
Here are the execution logs for when I run them in sequence:
Bug checkstyle-5 (fixed) failed to test
Bug gson-9 (buggy) passed tests
Bug rocketmq-9 (buggy) passed tests
Bug rocketmq-1 (buggy) passed tests
Bug openapi-generator-1 (buggy) passed tests
Bug jackson-core-1 (buggy) passed tests
Bug jackson-databind-2 (buggy) passed tests
Bug openapi-generator-2 (buggy) passed tests
Bug openapi-generator-5 (buggy) passed tests
Bug fastjson-1 (buggy) passed tests
Bug nacos-3 (buggy) passed tests
Bug checkstyle-12 (fixed) failed to test
Bug retrofit-1 (buggy) passed tests
Bug checkstyle-3 (fixed) failed to test
Bug checkstyle-6 (fixed) failed to test
Bug sslcontext-kickstart-4 (buggy) passed tests
Bug jackson-databind-3 (buggy) passed tests
For example, for "checkstyle-12" fixed:
TEST: com.puppycrawl.tools.checkstyle.checks.javadoc.InvalidJavadocPositionCheckTest
Failure/Error info:
InvalidJavadocPositionCheckTest.testDefault:93->AbstractModuleTestSupport.verifyWithInlineConfigParser:237->AbstractModuleTestSupport.verify:253->AbstractModuleTestSupport.verify:291->AbstractModuleTestSupport.verify:313->AbstractModuleTestSupport.verify:343->AbstractModuleTestSupport.lambda$verify$1:346 Violations for /tmp/elleelleaime/checkstyle-12-fixed-1c79e3a1-10f8-4d81-a9fe-42904f02c119/src/test/resources/com/puppycrawl/tools/checkstyle/checks/javadoc/invalidjavadocposition/InputInvalidJavadocPosition.java differ.
missing (5)
#1 : 73:6: Javadoc comment is placed in the wrong location.
#2 : 76:24: Javadoc comment is placed in the wrong location.
#3 : 79:43: Javadoc comment is placed in the wrong location.
#4 : 82:69: Javadoc comment is placed in the wrong location.
#5 : 94:1: Javadoc comment is placed in the wrong location.
unexpected (1)
#1 : 68:1: Javadoc comment is placed in the wrong location.
---
expected: [7:9: Javadoc comment is placed in the wrong location., 10:1: Javadoc comment is placed in the wrong location., 13:1: Javadoc comment is placed in the wrong location., 16:5: Javadoc comment is placed in the wrong location., 21:5: Javadoc comment is placed in the wrong location., 24:5: Javadoc comment is placed in the wrong location., 34:9: Javadoc comment is placed in the wrong location., 35:17: Javadoc comment is placed in the wrong location., 36:17: Javadoc comment is placed in the wrong location., 46:10: Javadoc comment is placed in the wrong location., 47:19: Javadoc comment is placed in the wrong location., 48:19: Javadoc comment is placed in the wrong location., 49:21: Javadoc comment is placed in the wrong location., 50:23: Javadoc comment is placed in the wrong location., 51:23: Javadoc comment is placed in the wrong location., 54:1: Javadoc comment is placed in the wrong location., 59:7: Javadoc comment is placed in the wrong location., 60:36: Javadoc comment is placed in the wrong location., 63:9: Javadoc comment is placed in the wrong location., 64:9: Javadoc comment is placed in the wrong location., 65:9: Javadoc comment is placed in the wrong location., 73:6: Javadoc comment is placed in the wrong location., 76:24: Javadoc comment is placed in the wrong location., 79:43: Javadoc comment is placed in the wrong location., 82:69: Javadoc comment is placed in the wrong location., 94:1: Javadoc comment is placed in the wrong location.]
but was : [7:9: Javadoc comment is placed in the wrong location., 10:1: Javadoc comment is placed in the wrong location., 13:1: Javadoc comment is placed in the wrong location., 16:5: Javadoc comment is placed in the wrong location., 21:5: Javadoc comment is placed in the wrong location., 24:5: Javadoc comment is placed in the wrong location., 34:9: Javadoc comment is placed in the wrong location., 35:17: Javadoc comment is placed in the wrong location., 36:17: Javadoc comment is placed in the wrong location., 46:10: Javadoc comment is placed in the wrong location., 47:19: Javadoc comment is placed in the wrong location., 48:19: Javadoc comment is placed in the wrong location., 49:21: Javadoc comment is placed in the wrong location., 50:23: Javadoc comment is placed in the wrong location., 51:23: Javadoc comment is placed in the wrong location., 54:1: Javadoc comment is placed in the wrong location., 59:7: Javadoc comment is placed in the wrong location., 60:36: Javadoc comment is placed in the wrong location., 63:9: Javadoc comment is placed in the wrong location., 64:9: Javadoc comment is placed in the wrong location., 65:9: Javadoc comment is placed in the wrong location., 68:1: Javadoc comment is placed in the wrong location.]
------------------------------------------------------------------------
Do you have any idea why this is happening?
Hm... this appears more difficult to answer than the previous one 😅 nonetheless, thanks for bringing this up :) I've asked the first author what he thinks of this, but because this is not an issue we were previously aware of, it might take a bit more time to resolve. We'll get back to you when we have a grasp of what is going on with those cases.
Thanks a lot!
Let me know if you need more information from me.
Hi, thanks for your interest in our work!
Here's what I have tried/found out:
-
Adding a similar line of script (binding another local directory to the container) to our repo libro/GHRB did not change the output of our bugs, which means that we cannot really reproduce your issue... Since our docker script already binds the working directory to the container, perhaps adding the sample files directly to the GHRB working directory might solve your problem?
-
Similarly, your forked repo ASSERT-KTH/GHRB worked fine when cloned into our server. This could mean that the version of your docker somehow crashes when launched for the repo. The docker version of our server is Docker version 20.10.7, build f0df350.
-
When testing all of the bugs again, we found out that the fixed version of our Checkstyle bugs are providing unexpected outputs, as in the example you provided. We will fix this issue as soon as possible, and we'd like to thank you for acknowledging such issue early.
I'll keep this thread open, and let you know when we have fixed the Checkstyle issue mentioned above.
Please let us know if you still face the same issue after trying our solutions.
Hi, Checkstyle bugs have been verified, and both of their versions should work fine now.
I'll close this issue now as there has been no further activity.