The only one failed test case is run twice in the fault localization stage in Nopol repair with Gzoltar localizer
DehengYang opened this issue · 14 comments
Sorry for my frequent disturbing. Your help and support is really appreciated.
I would like to share my doubts here. When I run the Nopol to repair Chart_3 from defects4j with the Gzoltar localizer, I found that the number of failed test cases are 2. However, when I run defects4j test in Chart 3, there is only one failed test case, the output is:
Failing tests: 1
org.jfree.data.time.junit.TimeSeriesTests::testCreateCopy3
So does the Nopol found more failed cases than as expected? Actually not. And I found that the two failed test cases obtained by Nopol are all "org.jfree.data.time.junit.TimeSeriesTests#testCreateCopy3". In other words. this only failed test case are run twice in the fault localization.
So why was this only failed test case run twice in one repair? As if it is run twice rather than once, the executed number of failed test cases will change to 2, which will directly lead to the change in the statements' suspiciousness. That is why I think it is important to figure this out.
What's more, the false test case "org.joda.time.tz.TestCompiler#testDateTimeZoneBuilder" was ran three times in Time_11, and so do the other Time bugs.
Thank you!
Thank you for your reply!
The version I am using is the Master version. When I try the "--flocal ochiai" flag, I found that for Chart programs, the Ochiai localizer didn't work and the output is:
So there is no patch result for these programs. I still don't figure out why this happens while ochiai is used.
For Time programs from Defects4j, the Ochiai fault localizer got the same result as Gzoltar. For instance, in Time_4, both fault localization results about the failed test cases are:
The number of failed test cases:
org.joda.time.TestPartial_Basics#testWith3 false
org.joda.time.TestPartial_Basics#testWith3 false
org.joda.time.TestPartial_Basics#testWith3 false
junit.framework.TestSuite$1#warning false
As is indicated by the results, the test case "testWith3" was run three times in the fault localization， with Ochiai or Gzoltar. I think this will have some uncertain impact on the accuracy of suspiciousness of statements, and this impact should not be neglected.
Thank you!
When I try the "--flocal ochiai" flag, I found that for Chart programs, the Ochiai localizer didn't work
So there is no patch result for these programs. I still don't figure out why this happens while ochiai is used.
This is a known bug, see #148, it would be great if you can help us to fix it.
As is indicated by the results, the test case "testWith3" was run three times in the fault localization， with Ochiai or Gzoltar
Interesting, thanks for the analysis. So it suggests it is a bug in Nopol, not in Gzoltar. Would you be able to debug, find the root cause and fix it?
Thanks!
Sure, I am highly interested in this issue and I will try to fix it. Thank you!
I'm thinking of several causes for this issue.
- our test case finder is buggy and return several times the same test class
- the tests are executed by several "ways", e.i. by the test-suite and individually (compatible JuniT 3 and Junit4)
- Gzoltar is buggy
@DehengYang have you identified the problem here?
I have thought and tried for a period of time last year, but after that I was stuck in other affairs... I am quite sorry for failing to solve this bug... But I will keep concerning the issue and trying to figure it out. Thank you!
And I found that the two failed test cases obtained by Nopol are all "org.jfree.data.time.junit.TimeSeriesTests#testCreateCopy3". In other words. this only failed test case are run twice in the fault localization.
I see that TimeSeriesTests is also run as part of the test suite in DataTimePackageTests, which means that it is run twice. It's probably the reason.
One solution is to manually remove DataTimePackageTests before starting Nopol.
Closing the issue.
Thank you for being so patient and nice in answering this question!
Hi @DehengYang !
Reading your papers "An Empirical Study on the Usage of Fault Localization in Automated Program Repair" and "An Empirical Study on the Effect of Dynamic Slicing on Automated Program Repair Efficiency".
Congrats for them!
--Martin
Dear Dr. Monperrus,
Thank you so much for your congratulations. You really help me a lot during this process.
Thank you again for your great help, patience, generosity and the cool and significant contributions you have made for the APR community!
Thanks for your contributions to the field!