damianszczepanik/maven-cucumber-reporting

Assume.assumeTrue(false); is showing fail in report

ishubhamjain opened this issue · 7 comments

ishubhamjain commented

I have some scenarios where if test data is not available for any scenario then we are skipping it beside of marking them failure

so I am using Assume.assumeTrue(false) of junit; which is showing skip in default report correctly
But Extendable reports is showing them as fail

Showing fail in Extendable reports

failtest

Showing skip in Default reports

skiptest

Example code (scenario 2):

import org.junit.Assume;

	@Given("^setup$")
	public void prerequisites() throws Throwable {
		System.out.println("setup");
		Assume.assumeTrue(false);
	}

<!-- https://mvnrepository.com/artifact/net.masterthought/maven-cucumber-reporting -->
<dependency>
    <groupId>net.masterthought</groupId>
    <artifactId>maven-cucumber-reporting</artifactId>
    <version>4.2.4</version>
</dependency>

please let me know if you require more details regarding same

damianszczepanik commented

I don't understand what is expectation and what the problem

ishubhamjain commented

@damianszczepanik thanks for your response. the expectation is when we use Assume.assumeTrue(false); instead of Assert of junit the test should be mark as skip in report. In default cucumber report as I shown in post the test mark in blue means it is mark as skip while in our maven-cucumber-reporting it is showing in red means it is mark as fail, Similarly when the count shown in the dashboard in report it also mark in skip not in fail

For an example:

capture

example code:

	@Given("^is setup$")
	public void example1() throws Throwable {
		System.out.println("is setup");
		Assume.assumeTrue(false);
		System.out.println("is setup ends)))))))))))))))))))))))))))))))))))))))))))))");
	}
	
	@Given("^is setup2$")
	public void example2() throws Throwable {
		System.out.println(" is setup");
		System.out.println(" is setup ends)))))))))))))))))))))))))))))))))))))))))))))");
	}
	
	@Given("^is setup3$")
	public void example3() throws Throwable {
		System.out.println("is setup");
		Assert.assertTrue(false);
		System.out.println("is setup ends)))))))))))))))))))))))))))))))))))))))))))))");
	}

In above dashboard you can see there is 3 cases out of it report showing 2 fail.
In above code you can three implementation.
Now example1() should be mark as skip, example2() should be mark as pass and example3() should be mark as fail. This is because example1() is using Assume class and example3() is using Assert.

So report should mark 1 fail scenario, 1 skip scenario and 1 fail scenario.
If you refer the cucumber default report screen shot, the scenario itself is mark as blue means skip not just its respective steps.

Please let me know your thought on same

damianszczepanik commented

Attach JSON file

ishubhamjain commented

@damianszczepanik - Please find the JSON below:

[
{
"line": 1,
"elements": [
{
"comments": [
{
"line": 4,
"value": "#@setup"
}
],
"line": 5,
"name": "Generating paper test 2",
"description": "",
"id": "paper",
"after": [
{
"result": {
"duration": 479691204,
"status": "passed"
},
"match": {
"location": "StepDefinition.afterFunction(Scenario)"
}
}
],
"type": "scenario",
"keyword": "Scenario",
"steps": [
{
"result": {
"duration": 415933450,
"status": "passed"
},
"line": 7,
"name": "is setup2",
"match": {
"location": "test4.prerequisites22()"
},
"keyword": "Given "
},
{
"result": {
"duration": 98196,
"status": "passed"
},
"line": 8,
"name": "Paper confirms",
"match": {
"location": "test4.prerequisites2()"
},
"keyword": "And "
},
{
"result": {
"duration": 313517,
"status": "passed"
},
"line": 9,
"name": "A existing",
"match": {
"location": "test4.prerequisites3()"
},
"keyword": "When "
},
{
"result": {
"duration": 286908,
"status": "passed"
},
"line": 10,
"name": "Paper confirmation",
"match": {
"location": "test4.prerequisites4()"
},
"keyword": "Then "
}
]
},
{
"line": 13,
"name": "Generating paper test",
"description": "",
"id": "paper-confirmation-generation-for-various-products;generating-paper-test",
"after": [
{
"result": {
"duration": 30343591,
"status": "passed"
},
"match": {
"location": "StepDefinition2.afterFunction(Scenario)"
}
}
],
"type": "scenario",
"keyword": "Scenario",
"steps": [
{
"result": {
"duration": 14554923,
"error_message": "org.junit.AssumptionViolatedException: got: \u003cfalse\u003e, expected: is \u003ctrue\u003e\r\n\tat org.junit.Assume.assumeThat(Assume.java:95)\r\n\tat org.junit.Assume.assumeTrue(Assume.java:41)\r\n\tat com.applications.eventgeneratorlot.test4.prerequisites(test4.java:28)\r\n\tat ✽.Given is setup(features/test3/01Paper.feature:15)\r\n",
"status": "pending"
},
"line": 15,
"name": "is setup",
"match": {
"location": "test4.prerequisites()"
},
"keyword": "Given "
},
{
"result": {
"status": "skipped"
},
"line": 16,
"name": "Paper confirms",
"match": {
"location": "test4.prerequisites2()"
},
"keyword": "And "
},
{
"result": {
"status": "skipped"
},
"line": 17,
"name": "A existing ",
"match": {
"location": "test4.prerequisites3()"
},
"keyword": "When "
},
{
"result": {
"status": "skipped"
},
"line": 18,
"name": "Paper confirmation",
"match": {
"location": "test4.prerequisites4()"
},
"keyword": "Then "
}
]
},
{
"line": 21,
"name": "Generating paper test 3",
"description": "",
"id": "paper",
"after": [
{
"result": {
"duration": 34549992,
"status": "passed"
},
"match": {
"location": "StepDefinition.afterFunction(Scenario)"
}
}
],
"type": "scenario",
"keyword": "Scenario",
"steps": [
{
"result": {
"duration": 5064750,
"error_message": "java.lang.AssertionError\r\n\tat org.junit.Assert.fail(Assert.java:86)\r\n\tat org.junit.Assert.assertTrue(Assert.java:41)\r\n\tat org.junit.Assert.assertTrue(Assert.java:52)\r\n\tat com.applications.eventgeneratorlot.test4.prerequisites222(test4.java:42)\r\n\tat ✽.Given is setup3(features/test3/01Paper.feature:23)\r\n",
"status": "failed"
},
"line": 23,
"name": "is setup3",
"match": {
"location": "test4.prerequisites222()"
},
"keyword": "Given "
},
{
"result": {
"status": "skipped"
},
"line": 24,
"name": "Paper confirms",
"match": {
"location": "test4.prerequisites2()"
},
"keyword": "And "
},
{
"result": {
"status": "skipped"
},
"line": 25,
"name": "A existing ",
"match": {
"location": "test4.prerequisites3()"
},
"keyword": "When "
},
{
"result": {
"status": "skipped"
},
"line": 26,
"name": "Paper confirmation",
"match": {
"location": "test4.prerequisites4()"
},
"keyword": "Then "
}
]
}
],
"name": "Paper confirmation generation for various products",
"description": "",
"id": "paper-confirmation-generation-for-various-products",
"keyword": "Feature",
"uri": "features/test3/01PaperConfirmation.feature"
}
]

damianszczepanik commented

Scenario is marked as passed when each step is passed, otherwise scenario is failed. This is by design

ishubhamjain commented

@damianszczepanik - If possible can we add this as an new feature for future releases as it is showing different stats and representation as per cucumber default reports

Additionally the cucumber JSON also marked it as skip.
Refer:

    "type": "scenario",
    "keyword": "Scenario",
    "steps": [
      {
        "result": {
          "duration": 14554923,
          "error_message": "org.junit.AssumptionViolatedException: got: \u003cfalse\u003e, expected: is \u003ctrue\u003e\r\n\tat org.junit.Assume.assumeThat(Assume.java:95)\r\n\tat org.junit.Assume.assumeTrue(Assume.java:41)\r\n\tat com.citi.oasys.confirmations.applications.eventgeneratorlot.test4.prerequisites(test4.java:28)\r\n\tat ✽.Given is setup(features/test3/01PaperConfirmation.feature:15)\r\n",
          "status": "pending"
        },
        "line": 15,
        "name": "is setup",
        "match": {
          "location": "test4.prerequisites()"
        },
        "keyword": "Given "
      },
      {
       "result": {
          "status": "skipped"
        },

Result is marked as skipped.

    "result": {
          "status": "skipped"
        },
damianszczepanik commented

I don't want to change it