Scott Test Reporter
Scott provides detailed failure messages for tests written in Java, without the use of complex assertion libraries to aid developers in rapid development, troubleshooting and debugging of tests. All information is presented on the source code of the test method as comments.
test_1(hu.advancedweb.example.CounterTest) Time elapsed: 0.005 sec <<< FAILURE!
java.lang.AssertionError:
9| @Test
10| public void test_1() {
11| Counter counter = new Counter(); // counter=Counter [state=0]
12|
13| counter.increase(); // counter=Counter [state=1]
14| counter.increase(); // counter=Counter [state=2]
15|
16| int state = counter.get(); // state=2
17|
18|* assertEquals(state, 3); // AssertionError: expected:<2> but was:<3>
19| }
at hu.advancedweb.example.CounterTest.test_1(CounterTest.java:18)
It automatically tracks the internal state of the tests to provide important details for a failing scenario, favoring simple assertions expressed mostly in plain Java over the extensive use of test libraries, such as Hamcrest or AssertJ - although Scott plays nicely with other testing tools and frameworks.
Scott does not intend to be a testing framework, nor does it provide an API to use in the tests. Instead, it aims to be a small tool that can be dropped into a project to do its job automatically, so you can worry much less about expressing assertions, and still have meaningful failure messages.
Scott: All systems automated and ready. A chimpanzee and two trainees could run her.
Kirk: Thank you, Mr. Scott. I'll try not to take that personally.
Supports JUnit 4, JUnit 5, and Cucumber JVM, on Java 7, Java 8, and Java 9.
How to use
After including Scott to the build flow, it automatically creates the detailed failure messages for failing tests. All you have to do is to write tests in Java with simple assertions or using your favorite testing library and run them as you would do normally. Scott will do its magic behind the scenes.
The recommended way to use Scott is the Scott Maven Plugin or Scott Gradle Plugin. With the former just add the following to your pom.xml
file. (See a minimal example pom file here.)
<build>
<plugins>
<!-- Add the Scott Plugin. -->
<plugin>
<groupId>hu.advancedweb</groupId>
<artifactId>scott-maven-plugin</artifactId>
<version>3.3.0</version>
<executions>
<execution>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
<dependencies>
<!-- Add Scott as a dependency -->
<dependency>
<groupId>hu.advancedweb</groupId>
<artifactId>scott</artifactId>
<version>3.3.0</version>
<scope>test</scope>
</dependency>
</dependencies>
If you prefer use Gradle, you can use our plugin (See a minimal example build.gradle file here.):
plugins {
id "hu.advanceweb.scott-plugin" version "3.3.0"
}
For complete examples see the following:
For a quick demo check out and mvn install
one of these example projects, as they not only
contain the required setup configuration (see the pom.xml
files), but a bunch of failing tests for the show.
For configuration options and alternative ways of using Scott, see this guide.
Cucumber
Scott for Cucumber tracks whole scenarios, and in case of a failure it prints the details of every step involved.
This feature provides valuable information if a test fails in a CI environment, as it can make it much easier to reproduce and fix browser-based tests, especially for flaky tests.
See the Cucumber JVM example project for details.
Features
Consider this failing test case:
@Test
public void myTest() {
Integer[] myArray = new Integer[] { 1, 4, 2, 4 };
List<Integer> myList = Arrays.asList(myArray);
Set<Integer> mySet = new HashSet<>(myList);
mySet.remove(4);
assertTrue(mySet.contains(4));
}
Normally it just produces an assertion error without a meaningful message. But with Scott, it shows additional information:
myTest(hu.advancedweb.example.ListTest) FAILED!
22| @Test
23| public void myTest() {
24| Integer[] myArray = new Integer[] { 1, 4, 2, 4 }; // myArray=[1, 4, 2, 4]
25| List<Integer> myList = Arrays.asList(myArray); // myList=[1, 4, 2, 4]
26|
27| Set<Integer> mySet = new HashSet<>(myList); // mySet=[1, 2, 4]
28| mySet.remove(4); // mySet=[1, 2]
29|
30|* assertTrue(mySet.contains(4)); // AssertionError
31| }
Notice that even without sophisticated assertions the required information is present in the test report.
For every failing test, Scott reports the
- assignments to local variables,
- changes made to objects referenced by local variables and fields,
- input parameters and
- relevant fields that the test accesses, but does not modify.
All information is nicely visualized on the source code of the test method.
Changelog
See Releases.
Highlights from the latest releases:
- Java 9 support
- Gradle Plugin
- Maven Plugin
- Scott Cucumber JVM now records whole scenarios
- Customizable tracking behavior
Contributing
Contributions are welcome! Please make sure to visit the contribution and development guide for some important notes on how to build and debug Scott. If you are looking for issues that can get you started with the development, see Issues marked with the help-wanted tag.