Multiple tags execution Issue from maven
Naumansh opened this issue · 20 comments
Summary
Old reference: #124
I amf facing the problem as described below:
1- ${tags} works fine with single scenario but doesn't work if multiple tags are passed like -Dtags="@validLogin2 and @validLogin3",it leads to generation of runners without execution.
2- If i remove ${tags} and directly try to run the test using cucumber expression like -Dcucumber.options="--tags @validLogin2" , it leads to generation of runners without execution.
Pom.xml
<groupId>com.trivago.rta</groupId>
<artifactId>cucable-plugin</artifactId>
<version>${cucable.version}</version>
<executions>
<execution>
<id>generate-test-resources</id>
<phase>generate-test-resources</phase>
<goals>
<goal>parallel</goal>
</goals>
</execution>
</executions>
<configuration>
<sourceRunnerTemplateFile>src/test/resources/configuration/CustomCreator.java</sourceRunnerTemplateFile>
<sourceFeatures>src/test/resources/features</sourceFeatures>
<generatedFeatureDirectory>${generated.feature.directory}</generatedFeatureDirectory>
<generatedRunnerDirectory>${generated.runner.directory}</generatedRunnerDirectory>
<parallelizationMode>scenarios</parallelizationMode>
<includeScenarioTags>${tags}</includeScenarioTags>
<logLevel>minimal</logLevel>
</configuration>
</plugin>```
Hi! Did you mistype -Dtags=-Dtags="@validLogin2 and @validLogin3
? Because this sets the tags property to -Dtags="@validLogin2 and @validLogin3
.
@laxersaz thats a type . sorry about that. updated !!
I have some questions:
- Do the generated runners look ok? Because if they do and they are not being executed, this is a problem of your test framework or failsafe configurations.
- What happens if you don't pass any tags? Are the generated runners executed?
- Do you actually have any scenarios that are tagged with
@validLogin2
AND@validLogin3
?
--> answering your question from the implemented framework instead of a sample one as below:
Do the generated runners look ok? Because if they do and they are not being executed, this is a problem of your test framework or failsafe configurations.
command: mvn clean verify -Dtags="@validateChangeinTaskMenuSequence-gHome and @validateSidebarTaskMenuRemoval-gHome"
So here is the console log as no runner is generated
[INFO] - includeScenarioTags : @validateChangeinTaskMenuSequence-gHome and @validateSidebarTaskMenuRemoval-gHome
[INFO] - parallelizationMode : scenarios
[INFO] - numberOfTestRuns : 1
[INFO] -------------------------------------
[INFO] - generated 0 scenarios from src/test/resources/features/login.feature.
[INFO] - generated 0 scenarios from src/test/resources/features/pagelayout.feature.
[INFO] - generated 0 scenarios from src/test/resources/features/phaseconfig.feature.
[INFO] - generated 0 scenarios from src/test/resources/features/stepprogressbar.feature.
[INFO] - generated 0 scenarios from src/test/resources/features/enhancedflexgrid.feature.
[INFO] - generated 0 scenarios from src/test/resources/features/expressionbuilder.feature.
[INFO] - generated 0 scenarios from src/test/resources/features/forms.feature.
[INFO] - generated 0 scenarios from src/test/resources/features/approvalprocess.feature.
[INFO] - generated 0 scenarios from src/test/resources/features/sidebar.feature.
[INFO] - generated 0 scenarios from src/test/resources/features/customApp.feature.
[INFO] - generated 0 scenarios from src/test/resources/features/charts.feature.
[INFO] - generated 0 scenarios from src/test/resources/features/flexgrid.feature.
[INFO] - generated 0 scenarios from src/test/resources/features/flextable.feature.
[INFO] - generated 0 scenarios from src/test/resources/features/calender.feature.
[INFO] -------------------------------------
[INFO] Cucable created 0 separate feature files and 0 runners.
Below is failsafe config:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>${maven.failsafe.plugin.version}</version>
<executions>
<execution>
<id>Run parallel tests</id>
<phase>integration-test</phase>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
<configuration>
<forkCount>${threads}</forkCount>
<testFailureIgnore>true</testFailureIgnore>
<disableXmlReport>true</disableXmlReport>
<argLine>-Dfile.encoding=UTF-8</argLine>
<excludes>
<exclude>**/IT*.java</exclude>
</excludes>
</configuration>
</plugin>
What happens if you don't pass any tags? Are the generated runners executed?
Yes the generation looks ok. Below console log for ref:
[INFO] - generated 87 scenarios from src/test/resources/features/pagelayout.feature.
[INFO] - generated 13 scenarios from src/test/resources/features/phaseconfig.feature.
[INFO] - generated 10 scenarios from src/test/resources/features/stepprogressbar.feature.
[INFO] - generated 110 scenarios from src/test/resources/features/enhancedflexgrid.feature.
[INFO] - generated 344 scenarios from src/test/resources/features/expressionbuilder.feature.
[INFO] - generated 8 scenarios from src/test/resources/features/forms.feature.
[INFO] - generated 31 scenarios from src/test/resources/features/approvalprocess.feature.
[INFO] - generated 10 scenarios from src/test/resources/features/sidebar.feature.
[INFO] - generated 6 scenarios from src/test/resources/features/customApp.feature.
[INFO] - generated 8 scenarios from src/test/resources/features/charts.feature.
[INFO] - generated 124 scenarios from src/test/resources/features/flexgrid.feature.
[INFO] - generated 67 scenarios from src/test/resources/features/flextable.feature.
[INFO] - generated 2 scenarios from src/test/resources/features/calender.feature.
Do you actually have any scenarios that are tagged with @validLogin2 AND @validLogin3?
Yes they did. Now sharing logs with real framework using tags @validateChangeinTaskMenuSequence-gHome AND @validateSidebarTaskMenuRemoval-gHome
In your first example, no scenarios seem to match. So naturally, there is nothing to run.
Yes. So what is the possible fix for it ?
Shouldn't the includeScenario accept multiple tags for execution as in documentation
<includeScenarioTags>@includeMe and @includeMeAsWell</includeScenarioTags>
Can you provide an example feature file for me to test this? Because without some data, it is hard for me to see if the problem is on your side or if it's a bug in Cucable
here is the feature file tests
@sidebar-component @regression
Feature: Side bar Component Scenarios
Test all sidebar component based scenarios
As a configuration manager i make different configurational changes in side bar component and will validate them.
@33877 @validateChangeinTaskMenuSequence-gHome
Scenario: validate after changing sequence of task menu from second to third in side bar
Given I have default configuration as below:
| configurationCategory | fieldName | fieldOrignalValue | filterId |
| Side Bar Header | Sidebar Component Sequence | Search,TaskMenu,ActivityMenu,RecentlyViewed,TechnicalSupport,Links | ApiGeneratedId:0 |
Given I setup test configuration as below:
| configurationCategory | fieldName | fieldNewValue | coreComponentField | coreCOmponentSubFieldName | coreComponentSubFieldValue |
| Side Bar Header:General | Sidebar Component Sequence | Search,ActivityMenu,TaskMenu,RecentlyViewed,TechnicalSupport,Links | GrantorHome | | |
When I login as "Automation" user
When I setup my app as "As a Grantor"
Then i see "Tasks" shown at number "4" in sideBar
@33876 @validateSidebarTaskMenuRemoval-gHome @smoke
Scenario: validate disabling of task menu from side bar
Given I have default configuration as below:
| configurationCategory | fieldName | fieldOrignalValue | filterId |
| Side Bar Header | Sidebar Component Sequence | Search,TaskMenu,ActivityMenu,RecentlyViewed,TechnicalSupport,Links | ApiGeneratedId:0 |
Given I setup test configuration as below:
| configurationCategory | fieldName | fieldNewValue | coreComponentField | coreCOmponentSubFieldName | coreComponentSubFieldValue |
| Side Bar Header:General | Sidebar Component Sequence | Search,ActivityMenu,RecentlyViewed,TechnicalSupport,Links | GrantorHome | | |
When I login as "Automation" user
When I setup my app as "As a Grantor"
Then I see "Tasks" as disabled from sidebar
You are asking Cucable to generate scenarios that match 2 tags. But none of your scenarios match your tag expression. So Cucable will generate no scenarios.
The and keyword means: match scenarios that include both of the provided tags only.
So if you want to match either tag, you need to use or.
-Dtags="@validateChangeinTaskMenuSequence-gHome and @validateSidebarTaskMenuRemoval-gHome"
or
-Dtags="@33877 and @33876"
I cannot understand how these two tag names are not matching ? i have these two tags on each of the above shared tests.
The first scenario has the tags @33877
and @validateChangeinTaskMenuSequence-gHome
.
The second scenario has @33876
, @validateSidebarTaskMenuRemoval-gHome
and @smoke
.
In your tag expression you are asking for @validateChangeinTaskMenuSequence-gHome and @validateSidebarTaskMenuRemoval-gHome
.
So you have no matches.
One test can have different tags which are independent of each other. Thats how it is always been.
Are you saying that if we have multiple tags on a test then it will be considered as consolidated 1 tag expression instead of individual tag ? It's a new thing for me though as we often have multiple tags for scenarios for grouping them during execution.
If that's so then can't we use each tag as separate tag ? isn't it a limitation ?
oh i think what you are saying that with and we can only specify tags of the same test ? right ?
Of course you can still use individual tags - I even told you how :-) . This is also stated in the documentation: https://github.com/trivago/cucable-plugin#includescenariotags
If you want to include scenarios that are tagged with validateChangeinTaskMenuSequence-gHome
or scenarios that are tagged with validateChangeinTaskMenuSequence-gHome
, you need to specify your tag expression like so
@validateChangeinTaskMenuSequence-gHome or @validateSidebarTaskMenuRemoval-gHome
The linked documentation section also has multiple examples for tag expressions and how to use them.
Exactly!
Exactly!
oh man...Thanks a lot. My bad i couldn't understand that it properly..Cheerz!!!
No worries :-)