Launch Open Test Alliance Initiative
sbrannen opened this issue · 36 comments
Status Quo
There is no standard for testing on the JVM: the only common building block we have is java.lang.AssertionError.
AssertionError is great for signaling that a test has failed, but it doesn't go far enough. Each testing framework is therefore forced to fill the gap with custom subclasses of AssertionError or RuntimeException to provide a richer feature set to end users. The downside is that each framework has its own set of custom errors and exceptions, and this makes it a challenge for frameworks to interoperate.
For example, JUnit has long supported the notion of a failed assumption via its AssumptionViolatedException, but assertion frameworks like AssertJ cannot integrate that feature without a direct dependency on JUnit. Furthermore, the status quo makes the work of IDEs and build tools more difficult than it should be.
Proposal
The only real solution to this problem is to create a foundation that we can all build on!
Based on recent discussions with IDE and build tool developers from Eclipse, Gradle, and IntelliJ, the JUnit Lambda team is working on a proposal for an open source project to provide a minimal common foundation for testing libraries on the JVM. The primary goal of the project is to enable testing frameworks like JUnit, TestNG, Spock, etc. and third-party assertion libraries like Hamcrest, AssertJ, etc. to use a common set of exceptions that IDEs and build tools can support in a consistent manner across all testing scenarios -- for example, for consistent handling of failed assertions and failed assumptions as well as visualization of test execution in IDEs and reports.
Draft Implementation
We have already begun with a small set of errors and exceptions that we consider to be common for all testing and assertion frameworks. In fact, we are already using these exceptions in the JUnit Lambda Prototype.
Please take a look at our current draft in the open-test-alliance project and let us know what you think.
Project Name and Repository
The name Open Test Alliance is simply a working title for the project. As such, the name is up for discussion, and recommendations are welcome. We will of course need a name that maps to an available Internet domain name so that the code can live in an appropriate Java package.
The draft implementation is currently a part of the JUnit Lambda repository; however, as soon as additional parties join the collaboration, we will collectively work to find a new home for the project.
Feedback is welcome!
What types of errors and exceptions should such a project support?
What types of properties should such errors and exceptions have?
What should the project be named?
Projects already contacted
We've already reached out to and asked for feedback from the maintainers of the following projects.
- Test NG
- Hamcrest
- AssertJ
- Spock
- Google Truth
- ScalaTest
- Eclipse
- IntelliJ
- Gradle
- Maven Surefire Plugin
I'd add that the name OTA is also up for discussion and that, as soon as there is more than one supporting party, the project will move into its own GitHub organisation.
Johannes
Am 17.11.2015 um 20:00 schrieb Sam Brannen notifications@github.com:
Status Quo
There is no standard for testing on the JVM: the only common building block we have is java.lang.AssertionError.
AssertionError is great for signaling that a test has failed, but it doesn't go far enough. Each testing framework is therefore forced to fill the gap with custom subclasses of AssertionError or RuntimeException to provide a richer feature set to end users. The downside is that each framework has its own set of custom errors and exceptions, and this makes it a challenge for frameworks to interoperate.
For example, JUnit has long supported the notion of a failed assumption via its AssumptionViolatedException, but assertion frameworks like AssertJ cannot integrate that feature without a direct dependency on JUnit. Furthermore, the status quo makes the work of IDE and build tools more difficult than it should be.
The Solution
The only real solution to this problem is to create a foundation that we can all build on!
Based on recent discussions with IDE and build tool developers from Eclipse, Gradle, and IntelliJ, the JUnit Lambda team is working on a proposal for an open source project to provide a minimal common foundation for testing libraries on the JVM. The primary goal of the project is to enable testing frameworks like JUnit, TestNG, Spock, etc. and third-party assertion libraries like Hamcrest, AssertJ, etc. to use a common set of exceptions that IDEs and build tools can support in a consistent manner across all testing scenarios -- for example, for consistent handling of failed expectations and failed assumptions as well as visualization of test execution in IDEs and reports.
Draft Implementation
We have already begun with a small set of errors and exceptions that we consider to be common for all testing and assertion frameworks. In fact, we are already using these exceptions in the JUnit Lambda Prototype.
Please take a look at our current draft in the open-test-alliance project and let us know what you think.
Feedback is welcome!
What types of errors and exceptions should such a project support?
What types of properties should such errors and exceptions have?
—
Reply to this email directly or view it on GitHub.
Very well done!
Sam Brannen notifications@github.com schrieb am Di., 17. Nov. 2015 um
21:38:
@jlink https://github.com/jlink, I agree, and I'll update the issue
accordingly.—
Reply to this email directly or view it on GitHub
#12 (comment)
.
Well written. Thanks.
Stefan Bechtold
Visitenkarte: http://ecard.bechte.de
http://www.bechte.de, http://twitter.com/bechte
Am 17.11.2015 um 20:00 schrieb Sam Brannen notifications@github.com:
Status Quo
There is no standard for testing on the JVM: the only common building block we have is java.lang.AssertionError.
AssertionError is great for signaling that a test has failed, but it doesn't go far enough. Each testing framework is therefore forced to fill the gap with custom subclasses of AssertionError or RuntimeException to provide a richer feature set to end users. The downside is that each framework has its own set of custom errors and exceptions, and this makes it a challenge for frameworks to interoperate.
For example, JUnit has long supported the notion of a failed assumption via its AssumptionViolatedException, but assertion frameworks like AssertJ cannot integrate that feature without a direct dependency on JUnit. Furthermore, the status quo makes the work of IDE and build tools more difficult than it should be.
The Solution
The only real solution to this problem is to create a foundation that we can all build on!
Based on recent discussions with IDE and build tool developers from Eclipse, Gradle, and IntelliJ, the JUnit Lambda team is working on a proposal for an open source project to provide a minimal common foundation for testing libraries on the JVM. The primary goal of the project is to enable testing frameworks like JUnit, TestNG, Spock, etc. and third-party assertion libraries like Hamcrest, AssertJ, etc. to use a common set of exceptions that IDEs and build tools can support in a consistent manner across all testing scenarios -- for example, for consistent handling of failed expectations and failed assumptions as well as visualization of test execution in IDEs and reports.
Draft Implementation
We have already begun with a small set of errors and exceptions that we consider to be common for all testing and assertion frameworks. In fact, we are already using these exceptions in the JUnit Lambda Prototype.
Please take a look at our current draft in the open-test-alliance project and let us know what you think.
Feedback is welcome!
What types of errors and exceptions should such a project support?
What types of properties should such errors and exceptions have?
—
Reply to this email directly or view it on GitHub.
Great news. :)
Is TestSkippedException intended a replacement/abstraction for AssumptionViolatedException?
@PascalSchumacher, almost... but not quite.
TestAbortedException is a replacement for AssumptionViolatedException:
Specialization of
IncompleteExecutionExceptionused to indicate that a test was aborted during execution.
TestSkippedException has slightly different semantics:
Specialization of
IncompleteExecutionExceptionused to indicate that a test was skipped prior to execution (e.g., disabled or ignored).
Hope that clarifies the difference.
Cheers,
Sam
Thanks for the clarification. Should have read the code and comments more carefully, but was to excited. :)
Excitement is a good thing!
Let's hope the community at large gets excited about pushing this initiative forward...
Well, @cbeust told me he was already contacted.
Maybe the list of contacted projects could help the community to not disturb the same persons many times :)
@juherr Thanks for your feedback! I've updated the issue description to include the projects we've already contacted.
Having spent a couple of weeks with TestSkippedException I tend to remove it from OTA. Since the alleged semantics are "Don't even start this test" it should never be thrown out of an already started test; how could we possibly prevent someone from doing that all the same?
As a matter of fact, all useful skipping in JUnit5 so far - @disabled and @conditional - gets along w/o throwing the exception.
Maybe some projects from https://github.com/atinfo/awesome-test-automation/blob/master/java-test-automation.md may be contacted too.
I like the idea. We definitely need a separate project soon.
What is the best way to provide feedback and discuss the actual project itself? Maybe we can setup a slack-chat or do we communicate via github issues only?
I'm not against Open Test Alliance just an alternative suggestion Common-Test-Framework/Foundation.
@jlink TestSkippedException would be useful for use in rules, test-listeners or other mechanisms that are executed before the actual test runs.
2015-11-19 17:34 GMT+01:00 Leonard Brünings notifications@github.com:
@jlink https://github.com/jlink TestSkippedException would be useful
for use in rules, test-listeners or other mechanisms that are executed
before the actual test runs.
Those rules would be framework specific since we don't have a common means
to introduce them. Thus they could use framework-specific means to announce
skipping of a test.
But maybe I'm lacking imagination. Could you sketch a scenario of a
framework independent rule?
If the meaning if TestSkippedException is "the test was entirely skipped" then it doesn't make sense for a Rule to throw it, since the test would have been partially executed (for example, previous Rules in the rule chain could have been executed).
I agree that we should remove it. We could always add it back later if there is a future need.
@jlink yes rules are Framework specific but the goal is to enable libraries to use a common set of constructs to express common situations and skipping a test would be one of them.
@kcooney yes you can argue that the execution has begun even if the test-body wasn't executed yet.
But as I said we need a different place to discuss these questions, we are already off-topic.
Guys, feel free to open a dedicated issue for the TestSkippedException discussion.
I've just created an ota tag to make it easier for us to find discussions about the OTA before it moves to a separate project.
AssertionFailedError
For infrastructure that runs tests and collects output, it's important that the expected and actual fields can be serialized. The JVM object will be gone after the test run, but the data is there to stay. ComparisonFailure had String-typed fields that could easily be serialized. Is the idea that tools use Object#toString() to store this data? If so, then this needs to be documented. And it begs the question why not to store Strings in the first place.
MultipleFailuresException
I'm not convinced this class is necessary. Since Java 7, java.lang.Throwable#addSuppressed(Throwable) can be used to report multiple exceptions. That also ensures the stacktraces of the additional exceptions are not lost.
MultipleFailuresException
I'm not convinced this class is necessary. Since Java 7,
java.lang.Throwable#addSuppressed(Throwable) can be used to report
multiple exceptions. That also ensures the stacktraces of the additional
exceptions are not lost.Can you elaborate on why a stack trace from an attached exception (when not
using addSuppressed) could get lost ? We were aware of addSuppressed but
considered it to be more explicit to have a dedicated exception for the
intended purpose of allowing more than one assert to give input to test
failure.
@mkeller We are still Java 6 compatible spockframework/spock#530 and I guess many other test frameworks still are, since they are used by other frameworks that are also still Java 6 compatible. If we enforce Java 7 here then Spock would support would be delayed to the next major version.
I wonder why you want to provide the implementation for AssertionFailedError, isn't interface enough? Agree on String instead of object, IDEs/build tools would most probably serialize them anyway and present default toString to the user if custom toString was not specified.
MultipleFailuresException: Exceptions usually get serialized via printStackTrace(...). The proposed MultipleFailuresException would only show messages from added failures, but not their stacktraces. If Java 6 compatibility is to be kept, then MultipleFailuresException definitely makes sense. In that case, MultipleFailuresException#printStackTrace(...) should also include the added failures.
Here's my feedback as an AssertJ developer and a (future) JUnit-lambda user, I haven't created issues for each points as they are related, let me know if you want me to do so and for which ones.
Also, I'm not sure all my points are relevant for OTA some might be for JUnit itself, I just thought they might be worth mentioning (just ignore them if they are not relevant).
1 - Support for String#format pattern when setting an assertion context
It should be easy to set an assertion context/description (which is done by calling as() in AssertJ), for example:
frodo.setAge(50);
assertThat(frodo.age).as("check %s's age", frodo.getName()).isEqualTo(33);gives the error message : [check Frodo's age] expected:<[33]> but was:<[50]>
As a user, I would not like to have to call String.format, that makes the assertion less readable and require more work.
2 - Support for String#format pattern when building the error message
This time I'm thinking more as an assertion library developer.
I often have to define messages using String.format, for example here's the template for a contains assertion error message:
"%nExpecting:%n <%s>%nto contain:%n <%s>%nbut could not find:%n <%s>%n", actual, expected, notFound
3 - Support multiline error message
It's sooooo much easier to understand the error when it is displayed on multiple lines, example
List<Ring> elvesRings = newArrayList(vilya, nenya, narya);
assertThat(elvesRings).containsOnly(nenya, vilya, oneRing);gives the error message:
Expecting:
<[vilya, nenya, narya]>
to contain only:
<[nenya, vilya, oneRing]>
elements not found:
<[oneRing]>
and elements not expected:
<[narya]>
4 - Automatic wrapping of collection display when too long
When displaying a collection, if the elements toString is long and the number of elements important it is hard to see what is going on, the library should be smart and wrap the collection display with one element per line instead one line for the all collection.
I can't resist to give an example using LotR ;-)
assertThat(fellowshipOfTheRing).contains(sauron);give's the error :
Expecting:
<[Frodo 33 years old Hobbit,
Sam 38 years old Hobbit,
Merry 36 years old Hobbit,
Pippin 28 years old Hobbit,
Gandalf 2020 years old Maia,
Legolas 1000 years old Elf,
Gimli 139 years old Dwarf,
Aragorn 87 years old Man,
Boromir 37 years old Man]>
to contain:
<[Sauron 50000 years old Maia]>
but could not find:
<[Sauron 50000 years old Maia]>
Mott of the previous points might be addresses by providing a fluent assertion error builder supporting multiline error message with String#format pattern (that would be great).
5 - Do not rely on toString only to display actual and expected
Example:
assertEquals(new Integer(10), new Long(10L));`fails with the not really helpful message :
expected:<10> but was:<10>
As Integer and Long are JDK types, JUnit should be able to display then differently, but what about user types ?
Well, the best we can do is display toString() and hashCode, at least it makes it clear that objects are not the same.
AssertJ
// Person's toString() method only display Person#name but not the age, equals checks both.
Person actual = new Person("Jake", 43);
Person expected = new Person("Jake", 47);
assertThat(actual).isEqualTo(expected);Assert Error message
Expecting:
<"Person[name=Jake] (Person@2325e7)">
to be equal to:
<"Person[name=Jake] (Person@232663)">
but was not.
JUnit
Assert.assertEquals(expected, actual);JUnit error message
expected:<Person[name=Jake]> but was:<Person[name=Jake]>
6 - Store enough information in the assertion error to help IDE show a visual diff.
You have already started that by storing actual and expected fields in AssertionFailedError, that's a good start, but we should make available to the IDE the relevant information explaining why an assertion failed, we could use a Diff object for that.
Let's see that on two collection assertions, I want to show that the diff informationi is highly assertion specific, we gonna use contains and containsOnly with the previous elves ring example:
contains assertion
List<Ring> elvesRings = newArrayList(vilya, nenya, narya);
assertThat(elvesRings).contains(nenya, vilya, oneRing);gives the error message:
Expecting:
<[vilya, nenya, narya]>
to contain:
<[nenya, vilya, oneRing]>
but could not find:
<[oneRing]>
What we should store in the assertion error given to the IDE:
- actual
- expected
- Diff would contain elements not found (
oneRing)
With this assertion error, an IDE could display the two collection and highlight the missing element.
The textual representation is still important for tests run from the command line.
containsOnly assertion
List<Ring> elvesRings = newArrayList(vilya, nenya, narya);
assertThat(elvesRings).containsOnly(nenya, vilya, oneRing);gives the error message:
Expecting:
<[vilya, nenya, narya]>
to contain only:
<[nenya, vilya, oneRing]>
elements not found:
<[oneRing]>
and elements not expected:
<[narya]>
What we should store in the assertion error given to the IDE:
- actual
- expected
- Diff would contain elements not found (
oneRing) and unexpected (narya)
With this assertion error, an IDE could display the two collection and highlight the missing element with one color and the unexpected with another color.
That's all for the moment ;-)
Hi Joel.
Many thanks for your detailed suggestions.
Your point #6 really requires a change to the exception to allow for
additional information.
As for the other points: Couldn't the all be completely handled in an
external library - as you do so well in AssertJ?
cheers, Johannes
2015-11-21 6:55 GMT+01:00 Joel Costigliola notifications@github.com:
Hi @sbrannen https://github.com/sbrannen and @jlink
https://github.com/jlinkHere's my feedback as an AssertJ developer and a (future) JUnit-lambda
user, I haven't created issues for each points as they are related, let me
know if you want me to do so and for which ones.Also, I'm not sure all my points are relevant for OTA some might be for
JUnit itself, I just thought they might be worth mentioning (just ignore
them if they are not relevant).
1 - Support for String#format pattern when setting an assertion contextIt should be easy to set an assertion context/description (which is done
by calling as() in AssertJ), for example:frodo.setAge(50);
assertThat(frodo.age).as("check %s's age", frodo.getName()).isEqualTo(33);gives the error message : [check Frodo's age] expected:<[33]> but
was:<[50]>As a user, I would not like to have to call String.format, that makes the
assertion less readable and require more work.
2 - Support for String#format pattern when building the error messageThis time I'm thinking more as an assertion library developer.
I often have to define messages using String.format, for example here's
the template for a contains assertion error message:"%nExpecting:%n <%s>%nto contain:%n <%s>%nbut could not find:%n <%s>%n", actual, expected, notFound
3 - Support multiline error message
It's sooooo much easier to understand the error when it is displayed on
multiple lines, exampleList elvesRings = newArrayList(vilya, nenya, narya);
assertThat(elvesRings).containsOnly(nenya, vilya, oneRing);gives the error message:
Expecting:
<[vilya, nenya, narya]>
to contain only:
<[nenya, vilya, oneRing]>
elements not found:
<[oneRing]>
and elements not expected:
<[narya]>4 - Automatic wrapping of collection display when too long
When displaying a collection, if the elements toString is long and the
number of elements important it is hard to see what is going on, the
library should be smart and wrap the collection display with one element
per line instead one line for the all collection.I can't resist to give an example using LotR ;-)
assertThat(fellowshipOfTheRing).contains(sauron);
give's the error :
Expecting:
<[Frodo 33 years old Hobbit,
Sam 38 years old Hobbit,
Merry 36 years old Hobbit,
Pippin 28 years old Hobbit,
Gandalf 2020 years old Maia,
Legolas 1000 years old Elf,
Gimli 139 years old Dwarf,
Aragorn 87 years old Man,
Boromir 37 years old Man]>
to contain:
<[Sauron 50000 years old Maia]>
but could not find:
<[Sauron 50000 years old Maia]>Mott of the previous points might be addresses by providing a fluent
assertion error builder supporting multiline error message with
String#format pattern (that would be great).
5 - Do not rely on toString only to display actual and expectedExample:
assertEquals(new Integer(10), new Long(10L));`
fails with the not really helpful message :
expected:<10> but was:<10>
As Integer and Long are JDK types, JUnit should be able to display then
differently, but what about user types ?
Well, the best we can do is display toString() and hashCode, at least it
makes it clear that objects are not the same.AssertJ
// Person's toString() method only display Person#name but not the age, equals checks both.Person actual = new Person("Jake", 43);Person expected = new Person("Jake", 47);
assertThat(actual).isEqualTo(expected);Assert Error message
Expecting:
<"Personname=Jake">
to be equal to:
<"Personname=Jake">
but was not.JUnit
Assert.assertEquals(expected, actual);
JUnit error message
expected:<Person[name=Jake]> but was:<Person[name=Jake]>
6 - Store enough information in the assertion error to help IDE show a
visual diff.You have already started that by storing actual and expected fields in
AssertionFailedError, that's a good start, but we should make available
to the IDE the relevant information explaining why an assertion failed, we
could use a Diff object for that.Let's see that on two collection assertions, I want to show that the diff
informationi is highly assertion specific, we gonna use contains and
containsOnly with the previous elves ring example:contains assertion
List elvesRings = newArrayList(vilya, nenya, narya);
assertThat(elvesRings).contains(nenya, vilya, oneRing);gives the error message:
Expecting:
<[vilya, nenya, narya]>
to contain:
<[nenya, vilya, oneRing]>
but could not find:
<[oneRing]>What we should store in the assertion error given to the IDE:
- actual
- expected
- Diff would contain elements not found (oneRing)
With this assertion error, an IDE could display the two collection and
highlight the missing element.
The textual representation is still important for tests run from the
command line.containsOnly assertion
List elvesRings = newArrayList(vilya, nenya, narya);
assertThat(elvesRings).containsOnly(nenya, vilya, oneRing);gives the error message:
Expecting:
<[vilya, nenya, narya]>
to contain only:
<[nenya, vilya, oneRing]>
elements not found:
<[oneRing]>
and elements not expected:
<[narya]>What we should store in the assertion error given to the IDE:
- actual
- expected
- Diff would contain elements not found (oneRing) and unexpected (narya
)With this assertion error, an IDE could display the two collection and
highlight the missing element with one color and the unexpected with
another color.That's all for the moment ;-)
—
Reply to this email directly or view it on GitHub
#12 (comment)
.
Johannes Link wrote to the Maven group. I provided my feedback to him and he asked if I could add it to the ticket. This is my feedback:
I took a look at JUnit 5 API. My only criticism is the @context annotation. I don't think developers should be encouraged to write inner classes for the sake of grouping tests. I believe this has already been sufficiently covered with the use of the new @tag and/or by method naming conventions (like testWithChild1, testWithChild2, etc.).
Furthermore, JEE has already has an established @context annotation in many of the EE specs which are associated with dependency injection. I think it's a bit jaring to see JUnit's @context be so different. My advice is to either drop @context (I don't like it's current meaning) or re-purpose it to be part of dependency injection -- possibly completely replacing the @nAmed injection and giving one common object to retrieve whatever you need to know about the test execution context/metadata.
Hi @pbenedict,
Thanks for providing feedback! However, it appears that you misunderstood our intentions.
This issue is about an open test alliance for defining common building blocks for all testing frameworks on the JVM. This issue has absolutely nothing to do with the proposed JUnit 5 API.
If you would like to comment on the programming or extension model for the JUnit Lambda prototype, please do so in a separate issue.
As an aside, the previous @Context annotation has since been renamed to @Nested.
Cheers,
Sam
I was thinking: Why not trying to involve the JCP and launch OTA under a JSR?
I don't have any experience with JCP, but it sounds like a year(s)-long
project, not like something we can pull off in a few weeks.
Johannes
2015-11-23 10:38 GMT+01:00 Julien Herr notifications@github.com:
I was thinking: Why not trying to involve the JCP and launch OTA under a
JSR?—
Reply to this email directly or view it on GitHub
#12 (comment)
.
+1 for the interface suggestion by @akozlova
Interfaces should be ok for any consumer but makes life much easier for implementors: They do not have to replace their own exception types by yours but just to implement the interface methods.
So the acceptance for joining OTA will be much better!
The problem with just defining interfaces is that you cannot catch an interface. You can only catch a subclass of Throwable.
You could of course catch Throwable and then perform instanceof checks on the interface types, but that leads to cumbersome code. In addition, explicitly catching Throwable isn't necessarily the best idea.
Furthermore, if the OTA only defined interfaces that would leave it up to the implementer to decide whether the actual exception is a checked exception, RuntimeException, Error, AssertionError, etc. In my opinion, this would make it difficult for testing frameworks to interoperate. We need not only a common interface but also standardized runtime semantics for throwing/catching such exceptions.
Of course, if I'm overlooking something, please enlighten me. ;)
Cheers,
Sam
I think we should leave conceptual and implementation details out of this thread, it should be about whether we think that OTA does have merrit and if we should start it as a project. Then and only then should we begin to talk about such details.
Hi everybody,
The OTA finally has its own home:
https://github.com/ota4j-team/opentest4j
Feel free to continue the discussions there.
In conjunction with issue #30, we will be making changes to the current draft based on feedback several of you have provided here.
Once issue #30 is resolved, we will discontinue discussions regarding the OTA in this GitHub repository unless such discussions pertain only to JUnit.
Speaking for google/truth, we're quite happy to adopt this as a de-facto standard once we can get a stable release artifact. I'm going to work up a provisional branch to integrate it.
Thanks for all of your feedback and enthusiasm!
Since #30 is now closed...
Please continue discussions in the official https://github.com/ota4j-team/opentest4j repository.