Issues with skipped tests in summary and html report on V0.9.2
Adarshkrishna9099 opened this issue · 11 comments
This is issue with respect to karate.abort() function where I am able to successfully skip the steps once my condition is met but where as I am able to see the status of the test as Failed in cucumber html reports.
sorry closing as you have not followed the instructions: https://github.com/intuit/karate/wiki/How-to-Submit-an-Issue
I also said in https://stackoverflow.com/questions/55944353/karate-abort-script-failing-in-version-0-9-2 that it's expected to have a fail when abort() is used.
@MercierAdrien I just added a demo example for this, see commit above. I think the maven cucumber report has logic to mark the scenario as failed if there are skipped steps - let me know if my understanding is right. in Karate, we treat it as a pass, which is reflected in the Results object when you use the Runner.
maybe there is an opportunity to influence the 3rd party reports, but I remember we experimented with different statuses in #416 #464 and finally decided on SKIPPED. I'm not in favor of changing it - unless someone can dig into this more. personally I have never liked abort() - you can get the same effect with a call and an eval if (condition) karate.call()
looks like this has been requested here: damianszczepanik/cucumber-reporting#636
but looks unlikely to be fixed in the near future. I'm proposing we flip any skipped steps after an abort to "passed" - well anyway I keep saying that abort() is not the most elegant way to write tests - so here goes. reopening
for the reference of those looking for more info the developer of the 3rd party report has changed the behavior, refer damianszczepanik/cucumber-reporting#862 (comment)
@ptrthomas : It looks like that Cucumber-reporting fixed the issue: damianszczepanik/cucumber-reporting#862
Would it make sense to undo this changes so Aborted-tests will get the status 'skipped' again?
@gerben86 yes. it requires some research into which version of that dependency fixes things (was not clear last I checked) and whether anything else breaks. this is one dependency that we have recommended the same old version for 2.5 years.
if anyone does the research and comes up with a recommendation, sure :)
@gerben86 how about this - we intro a * configure abortedStepsShouldPass = true option which defaults to false ? I am marking this issue as not-fixed
@ptrthomas : I think that could work, because then we can decide how to treat the abort().
implemented configure abortedStepsShouldPass which defaults to false in develop
released 0.9.5