UT-SE-Research/iDFlakies

Substantial performance difference between iDFlakies versions

winglam opened this issue · 1 comments

Between iDFlakies version 1.0.2 and 1.1.0-SNAPSHOT, some changes apparently cause iDFlakies to run much slower.

E.g., for alibaba/fastjson and iDFlakies version 1.0.2, running 55 tests take only 33s, while iDFlakies version 1.1.0-SNAPSHOT takes 4m47s. Both runs do not find any flaky tests.

Aliaba fastjson version: 65f65e1443484b91d427253e79ed305ca79d6d57
iDFlakies 1.1.0-SNAPSHOT: 2ea44c2

original-order.log
idflakies-1.1.0-SNAPSHOT.log
idflakies-1.0.2.log

The cause for the performance difference is because the support for JUnit 5 required a check to see if a project would contain a mix of JUnit 4 and JUnit 5 tests, or just one or the other.
https://github.com/idflakies/iDFlakies/blob/8c446cb7ba7a9708c10e5e7975bcda646e55a452/src/main/java/edu/illinois/cs/dt/tools/detection/DetectorPlugin.java#L219

No fix is deemed necessary as the check is needed to throw an exception if there are mixed tests.

Changes were introduced to prevent the locating of tests from running multiple times unnecessarily.
8c446cb