hcoles/pitest

PIT does not work with EvoSuite's separateClassLoader mode

eran505 opened this issue · 5 comments

eran505 commented

Hello,

Im currently working on Common-Math-3.5, i generate test suite using Evosuite-1.0.5.
As i run PIT on some classes, the program generate mutations and coverage just fine, but i have zero kills over the mutations.
I took one mutation (from the html file) and ran it manually using Intellij IDE, and the test failed ! but PIT marked it as survived.
So i guess PIT didnt invoked the test classes .

pom.xml

image

console
image
image

Html file
image

Thanks

hcoles commented

If you can upload a minimal project that reproduced the issue I can take look.

eran505 commented

Thanks for the fast response,
I uploaded a minimal project that demonstrate the issue.
Inside the project there are two test classes, one made by me, and one by Evosuite-1.0.5.

Both tests have the same test case, in The Stack_ESTest.java ( made by Evosuite ) "test0"
In StackTest.java made by me, the equivalent method to test0 is "test" (the only test case in this calss)
image

image

when i ran PIT with Stack_ESTest , the result is 0/10 kills
when i ran PIT with StackTest , the result is 4/10 kills

the link to the project: https://github.com/eran505/simple_proj.git
or : https://files.fm/u/36zhjffg

hcoles commented

The problem looks to be triggered by setting separateClassLoader = true in Evosuite.

I'd need to spend a bit of time debugging this to understand what this does and why it causes a problem, but the issue can be worked around in the short term by setting it to false.

The behaviour is strange. Pitest looks to be executing the code (it shows coverage), but not registering the asserts.

eran505 commented

Thanks Henry, after the change Pitest works on Evosuite test suite.

One last thing, i wonder if Pitest is deterministic, can you refer me to an article, or tell me in short details?

hcoles commented

Pitest is fully deterministic if the code and tests fed into it are deterministic with just one caveat.

Pitest uses test execution times for two purposes

  • Determining if an infinite loop has occurred
  • Picking the order in which to run tests against mutants

As timings can be influenced by external factors (ie other programs running on the box) this can cause slight differences between runs.