Still not able to generate new tests except adding assertions on XWiki

[vmassol]

The title says it all. Tested up till DSpot 2.1.0. Last experiment was with DSpot 2.2.0 but it failed to execute (see #878).

[danglotb]

Hi @vmassol

Do you open this issue because DSpot fails to be executed or because DSpot is not able to generate any amplified test methods that increase the mutation score?

If it the former reason, please give me more details, like the complete execution trace using the verbose mode of DSpot: <verbose>true</verbose>.

If it is the latter reason, I ran dspot on xwiki-commons-xml from a freshly cloned XWiki repos (commit 1082f466b5c5d3b9a039492cebc40bcc1dea727e), so the used version of DSpot is 2.2.0.

Ran from the module, as you do, I have the following output:

Test class that has been amplified: org.xwiki.xml.XMLUtilsTest
The original test suite kills 85 mutants
The amplification results with 0 new tests
it kills 0 more mutants

Test class that has been amplified: org.xwiki.xml.html.HTMLUtilsTest
The original test suite kills 85 mutants
The amplification results with 0 new tests
it kills 0 more mutants

Test class that has been amplified: org.xwiki.xml.internal.html.DefaultHTMLCleanerTest
The original test suite kills 85 mutants
The amplification results with 0 new tests
it kills 0 more mutants

Test class that has been amplified: org.xwiki.xml.html.filter.AbstractHTMLFilterTest
The original test suite kills 85 mutants
The amplification results with 0 new tests
it kills 0 more mutants

Test class that has been amplified: org.xwiki.xml.stax.StAXUtilsTest
The original test suite kills 85 mutants
The amplification results with 0 new tests
it kills 0 more mutants

So the amplification seems to be completed without any error, but without any success. Recalling that we are not using any Input amplifiers and descartes as test-criterion. Since descartes generates less mutants than gregor, it might be more difficult to increase the mutation score.

I obtain the exact same trace using dspot-2.2.1-SNAPSHOT.

It seems that there are some errors when trying to compute the mutation score of a specific amplified test class, i.e. org.xwiki.xml.html.filter.AbstractHTMLFilterTest:

18:04:44 PIT >> SEVERE : Description [testClass=org.xwiki.xml.html.filter.AbstractHTMLFilterTest, name=filterChildren()] did not pass without mutation.
18:04:44 PIT >> SEVERE : Description [testClass=org.xwiki.xml.html.filter.AbstractHTMLFilterTest, name=filterDescendants()] did not pass without mutation.
18:04:44 PIT >> SEVERE : Description [testClass=org.xwiki.xml.html.filter.AbstractHTMLFilterTest, name=filterDescendantsWithElementSelector()] did not pass without mutation.
18:04:44 PIT >> SEVERE : Description [testClass=org.xwiki.xml.html.filter.AbstractHTMLFilterTest, name=hasAttribute()] did not pass without mutation.
18:04:44 PIT >> SEVERE : Description [testClass=org.xwiki.xml.html.filter.AbstractHTMLFilterTest, name=moveChildren()] did not pass without mutation.
18:04:44 PIT >> SEVERE : Description [testClass=org.xwiki.xml.html.filter.AbstractHTMLFilterTest, name=replaceWithChildren()] did not pass without mutation.

I guess it is because it is abstract. But I doubt that it is the principal cause of the unsuccessful executions.

Let's have a look to the manual tests, because the potential of DSpot relies mostly on them:

In DefaultHTMLCleanerTest, most unit tests use a custom assertion method: assertHTML(String expected, String actual), meaning that the potential is considerably reduced.

In StAXUtilsTest and XMLUtilsTest, most of the tests are only using assertions and/or static objects. I think that kind of tests are not good candidate for DSpot.

HTMLUtilsTest has good candidate for the amplification done by DSpot, i.e. a clear input part and an oracle part.

However, even with input amplifiers, I'm afraid that the potential neighborhood is very small since there are not a lot of element on which DSpot can play.

Let's take an example:

@Test
public void stripHTMLEnvelope() throws Exception
{
    Document document =
    this.cleaner.clean(new StringReader("<html><head><body><p>test1</p><p>test2</p></body></html>"));
    HTMLUtils.stripHTMLEnvelope(document);
    assertEquals(DefaultHTMLCleanerTest.HEADER + "<html><p>test1</p><p>test2</p></html>\n",
    HTMLUtils.toString(document));
}

Here, we have a org.w3c.dom.Document. DSpot won't do anything since it is a third-party object.
There is the String literal that can be amplified: <html><head><body><p>test1</p><p>test2</p></body></html>.
And that's all, since HTMLUtils.stripHTMLEnvelope(document); is a void return method.

All of this to explain that the seed test method are really important.

If you have any question or suggestion, please do not hesitate.