pankti-experiments

Scripts, generated tests, and data by pankti

Project Selection Criteria

Regarding the experimental targets, the following criteria is applied:

The project should be a real-world product which has a certain number of end-users instead of a prototype/toy example
The project should contain a certain number of GitHub stars and commits, which indicate its popularity and complexity
The project can be deployed with the research lab's calculation resource, with a relatively realistic workload for experiments

There are 3 projects selected for the experiments. The following table shows the descriptive metrics of each project.

project	Version	LOC	Classes	App. Methods	Test Methods	TS Method Cov.	TS Line Cov.	WL Method Cov.	WL Line Cov.
jitsi-jicofo	stable-4857	Sum: 28,318 Java: 25,165	146	1,350	63	49.4% 667/1,350	46.7% 3,537/7,571	48.9% 660/1,350	46.2% 3,500/7,571
PDFBox	2.0.21	Sum: 728,812 Java: 161,976	1,278	11,042	1,675	54.8% 6,049/11,042	53.5% 34,653/64,787	21.6% 2,390/11,042	21.0% 13,630/64,787
Broadleaf	6.1.4-GA	Sum: 618,465 Java: 211,030	684	6,173	197	23.9% 1,478/6,173	23.7% 5,846/24,667	23.1% 1,424/6,173	19.2% 4,735/24,667

The Commands for Descriptive Metrics Measurement

lines of code

Command cloc (Count Lines of Code) is used to calculated LOC.

cloc .

Classes, TS Method Cov., and TS Line Cov.

Add the following xml code into the pom file of the target project within <project></project>.

<build>
  <plugins>
    <plugin>
      <groupId>org.jacoco</groupId>
      <artifactId>jacoco-maven-plugin</artifactId>
      <version>0.8.5</version>
      <executions>
        <execution>
          <id>default-prepare-agent</id>
          <goals>
            <goal>prepare-agent</goal>
          </goals>
        </execution>
        <execution>
          <id>default-report</id>
          <goals>
            <goal>report</goal>
          </goals>
        </execution>
      </executions>
    </plugin>
  </plugins>
</build>

Then run mvn test and mvn jacoco:report. The JaCoCo report will be generated at ./target/site/jacoco. If the project contains multiple modules, the plugin can be declared in the root pom file. Run mvn jacoco:report-aggregate after mvn test to get an aggregated report.

JaCoCo for Maven Multi-Module Builds

If a project contains multiple sub-modules, some extra work might be needed in order to generate an aggregated JaCoCo report. (https://github.com/jacoco/jacoco/wiki/MavenMultiModule)

Create a dedicated module in your project for generation of the report. This module should depend on all or some other modules in the project. Here is an example for PDFBox.
Run mvn jacoco:report-aggregate after mvn test to get an aggregated report.
If your project already defines VM arguments for test execution, be sure that they will include property defined by JaCoCo. (https://www.jacoco.org/jacoco/trunk/doc/prepare-agent-mojo.html)

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <configuration>
    <argLine>@{argLine} -your -extra -arguments</argLine>
  </configuration>
</plugin>

Another Workaround for Aggregated Report Generation

Simply run mvn test first, after which a set of jacoco.exec files should be generated in each sub-module's folder
Use jacococli.jar to merge all the execution data files, e.g.,

java -jar jacococli.jar merge sub-module-a/jacoco.exec sub-module-b/jacoco.exec --destfile jacoco-merged.exec

Use jacococli.jar to generate an aggregated report manually

java -jar jacococli.jar report ./jacoco-merged.exec \
--classfiles ./sub-module-a/target/classes \
--classfiles ./sub-module-b/target/classes \
--sourcefiles ./sub-module-a/src/main/java \
--sourcefiles ./sub-module-b/src/main/java \
--html ./jacoco-report

This is very straightforward, though a bit tedious.

WL (workload) Method Cov. and WL Line Cov.

Download JaCoCo agent jar from here: https://www.jacoco.org/jacoco/

Add the following options when starting the target application:

-javaagent:<path-to-jacoco>/lib/jacocoagent.jar

Then feed the application with the workload, the coverage information will be dumped into file jacoco.exec when the JVM terminates. Use the following command to generate an html report:

java -jar <path-to-jacoco>/lib/jacococli.jar report ./jacoco.exec --classfiles <path-to-application>/target/classes/com --sourcefiles <path-to-application>/src/main/java --html ./jacoco-report

Note that --classfiles and --sourcefiles can be specified multiple times if you want to analyze several specific sub-modules in the project.

ASSERT-KTH/pankti-experiments