Cucumber Scala final output shows 0 failures even when a scenario fails

[rhande-mdsol]

I have a cucumber test suite which has some tests

I run the tests using command

sbt "testOnly <cucumber_class_name>"

And one of the cucumber tests fail an assertion and console shows

Failed scenarios:
classpath:features/my_cucumber_scenario.feature:31# <my test description>

19 Scenarios (1 failed, 18 passed)
88 Steps (1 failed, 87 passed)sts 214s
3m18.224s

But in the end the console shows

Total 80, Failed 0, Errors 0, Passed 80

Since one of the tests failed, in the final output it should have shown
Failed 1 (Or something like that). But it is showing all passed instead due to which my build on travis is not failing.

How to solve this problem ? Is there a solution in cucumber ?

My dependencies are

"io.cucumber" % "cucumber-core" % "5.7.0",
"io.cucumber" % "cucumber-junit" % "5.7.0"
"io.cucumber" %% "cucumber-scala" % "5.7.0"

[gaeljw]

Hello @rhande-mdsol ,

Could it be that you have multiple scenarios with the same name (in the same file), or Scenario Outlines?

If so, this is is a known issue in sbt-junit interface:
sbt/junit-interface#84. See also #22

[mpkorstanje]

Might be worth adding a work around to detect if sbt-junit is used and act accordingly. The project seems dead.

[mpkorstanje]

Or ensure the names provided to the JUnit Description are always unique. That way sbt will also be able to handle identical features, scenarios, examples, ect. Shouldn't be too much of a burden if the #n suffix only shows up when necessary.

[mpkorstanje]

Would either of you mind checking if cucumber/cucumber-jvm#2045 solves the problem.

[rhande-mdsol]

Sorry Closed by mistake.

[rhande-mdsol]

@mpkorstanje I am using

"io.cucumber" % "cucumber-core" % "5.7.0",
"io.cucumber" % "cucumber-junit" % "5.7.0"
"io.cucumber" %% "cucumber-scala" % "5.7.0"

So it is cucumber scala. Your code in cucumber/cucumber-jvm#2045
seems to be Java code. Can I use it ?

I will try publishing your code and let you know

[gaeljw]

@rhande-mdsol Cucumber Scala is using Cucumber JVM which is the foundation for various Cucumber libraries targeting the JVM.

In this case, it's the project cucumber-junit that you could try to build on your own with the PR mentioned by @mpkorstanje and use in your project.

I will also try the PR on my side but probably later today.

---

@rhande-mdsol Do you confirm that your case is multiple scenarios with the same name in the same file, or using Scenario Outlines ?

[rhande-mdsol]

@rhande-mdsol Cucumber Scala is using Cucumber JVM which is the foundation for various Cucumber libraries targeting the JVM.

In this case, it's the project cucumber-junit that you could try to build on your own with the PR mentioned by @mpkorstanje and use in your project.

I will also try the PR on my side but probably later today.

@rhande-mdsol Do you confirm that your case is multiple scenarios with the same name in the same file, or using Scenario Outlines ?

Hi @gaeljw ,

Yes, this problem happens when I have a Scenario Outline: and then have a table like structure in Examples: section. For me 3rd example actually failed, but final result showed all tests passed.
Also this happens when you have a Step with parameterized value and reuse the step in the same Feature file at multiple places.

For example :

Scenario 1 =>

Scenario:
Given I know "Java"
Then I teach "Java"

Scenario 2 =>

Scenario:
Given I know "Scala"
Then I teach "Scala"

If Scenario 2 fails here, the final result still shows all have passed.

I also tried failing a scenario without an outline (i.e. Scenario:) in this case final result shows failures properly.

Thanks

[gaeljw]

Thanks for confirming.

cucumber/cucumber-jvm#2045 should fix the issue then.
You just have to wait a bit for the next cucumber-junit release (you'll have to update to 6.x but the upgrade from 5.x to 6.x is pretty easy).

[rhande-mdsol]

Thanks for fixing the problem. Now final result shows failure when there is a failure in scenario outline.

[ejaoua]

Hi @rhande-mdsol ,
can you please share what was the plugin and/or correct syntax that you used to run one test of Cucumber :
sbt "testOnly <cucumber_class_name>".
I tried the following but it didn't work out sbt "testOnly *classpath:features/folder1/endpoint.feature
Thanks.

[gaeljw]

@ejaoua you might want to take it to Slack to discuss these kind of things, or open a new issue/discussion at least.

Short answer: I don't think that's possible this way. You have to run testOnly SomeClass where SomeClass targets only some features file.