Test inclusion patterns are derived from application code
nrainer opened this issue · 10 comments
Characteristics
- Issue Type: bug
- Reproducibility: always
- Severity: major
- Execution Environment: Ubuntu x64
The problem is that test inclusion patterns are derived from the application code. (As a consequence, no tests from projects without application code are executed.)
Old ticket: Tests from modules without application code are skipped
Description
Modules that contain exclusively tests (and no source code) are skipped from the analysis. For example, some projects have a dedicated module for integration tests, which is skipped in this case. (Actually, the tests of this project get skipped, but the mutations analysis is conducted. A result file is created.) Hence, not all tests of a project will get executed.
Steps to reproduce
- Unzip the attached sample project. It is a multi-module project consisting of the modules
core
,extension
, andtests
. All modules contain tests, all modules excepttests
contain application code. The moduletests
does not contain application code and has nosrc/main/java
folder. - After building the project with
mvn clean install
, run pit-mp usingmvn eu.stamp-project:pitmp-maven-plugin:run
MyTest
from the moduletests
covers a class from the moduleextension
but does not get executed. See the attached mutation matrix.
Files
Is the problem probably caused by mpMojo#shouldRun
in line 214? (Line number of revision 72ebb6e)
Thanks for the test case and the PR @nrainer
Sorry for the delay, I'm trying to catch up... :-)
I hope I can release a new version by next Friday.
@CaelInria You are welcome :)
I think this issue can be closed since the corresponding PR has been merged.
@CaelInria I fear that this problem was reintroduced with 1.3.5. :-/
The point is that we need to explicitly set the patterns for the test cases because otherwise PIT will derive them from the application code (src/main/java) (see [1]). As a consequence, the patterns created by PIT will only match tests in packages that also exist in the application code. Hence, tests in packages that do not exist in the application code will not be considered. A special case are modules without application code, in which PIT will include no test (because no packages in the application code exist).
Therefore, updateTests
in PmpMojo
[2] is definitively needed in my opinion. :)
[1] org.pitest.mutationtest.config.ReportOptions.getTargetTestsFilter():300
[2] https://github.com/STAMP-project/pitmp-maven-plugin/blob/master/src/main/java/org/pitest/maven/PmpMojo.java#L340
@nrainer
Sorry, I probably miss something, but I tried on both dnoo and your example project-with-module-with-only-tests and I have a report for dnooIntegration (has only tests and no sources):
Is it the result your are expecting ?
I fear that it is possible to answer the question based on the HTML report. From what I understand, this report lists the code under test (i. e., application code) but provides no direct information about the executed tests.
However, from what I remember when I was debugging PIT to analyze this ticket, I am quite sure that the change is necessary...
Can you please check if the tests located in dnooIntegration
itself were executed (e.g., by using XML as output and finding a test there in killingTests
)? (Note that when processing dnooIntegration
tests from other modules might get executed.)
If so, does the package of these integration tests exist in one of the source folders of any module? If so, can you please rename the test package to a name that does not exist anywhere (and run mvn install
afterwards) and check again if the tests were executed? I assume they won't be executed.
Let me know if you need further information. :)
Probably easier:
- Create a multi-module-project with two modules A and B.
- Module A: add a simple class with one method in package
org.pit.x
. - let module B depend on module A
- Module B: add a test case that tests the class from module A (and can kill the mutations) and place it in the test source folder in package
org.pit.y
. - run the mutation analysis, check if the test is executed / if the mutations are killed. I think that the test won't be executed. --> NOT OK
- rename the package in module B to
org.pit.x
and rerunmvn install
- run the mutation analysis, check if the test is executed / if the mutations are killed. I think that the test will be executed. --> OK
I checked the values... this issue is corrected in v1.3.6 :-)
Cool, thanks for testing and adjusting! 👍