Multiple tags execution Issue from maven

Question

Multiple tags execution Issue from maven

Naumansh opened this issue 5 years ago · 20 comments

Summary
Old reference: #124
I amf facing the problem as described below:
1- ${tags} works fine with single scenario but doesn't work if multiple tags are passed like -Dtags="@validLogin2 and @validLogin3",it leads to generation of runners without execution.
2- If i remove ${tags} and directly try to run the test using cucumber expression like -Dcucumber.options="--tags @validLogin2" , it leads to generation of runners without execution.

Pom.xml

				<groupId>com.trivago.rta</groupId>
				<artifactId>cucable-plugin</artifactId>
				<version>${cucable.version}</version>
				<executions>
					<execution>
						<id>generate-test-resources</id>
						<phase>generate-test-resources</phase>
						<goals>
							<goal>parallel</goal>
						</goals>
					</execution>
				</executions>
				<configuration>
					<sourceRunnerTemplateFile>src/test/resources/configuration/CustomCreator.java</sourceRunnerTemplateFile>
					<sourceFeatures>src/test/resources/features</sourceFeatures>
					<generatedFeatureDirectory>${generated.feature.directory}</generatedFeatureDirectory>
					<generatedRunnerDirectory>${generated.runner.directory}</generatedRunnerDirectory>
					<parallelizationMode>scenarios</parallelizationMode>
					<includeScenarioTags>${tags}</includeScenarioTags>
					<logLevel>minimal</logLevel>
				</configuration>
			</plugin>```

bischoffdev commented 5 years ago

Exactly!

Answer 1 · 2019-07-17T12:00:23.000Z

Hi! Did you mistype -Dtags=-Dtags="@validLogin2 and @validLogin3? Because this sets the tags property to -Dtags="@validLogin2 and @validLogin3.

Answer 2 · 2019-07-17T12:24:42.000Z

@laxersaz thats a type . sorry about that. updated !!

Answer 3 · 2019-07-17T13:54:09.000Z

I have some questions:

Do the generated runners look ok? Because if they do and they are not being executed, this is a problem of your test framework or failsafe configurations.
What happens if you don't pass any tags? Are the generated runners executed?
Do you actually have any scenarios that are tagged with @validLogin2 AND @validLogin3?

Answer 4 · 2019-07-17T14:12:27.000Z

--> answering your question from the implemented framework instead of a sample one as below:

Do the generated runners look ok? Because if they do and they are not being executed, this is a problem of your test framework or failsafe configurations.

command: mvn clean verify -Dtags="@validateChangeinTaskMenuSequence-gHome and @validateSidebarTaskMenuRemoval-gHome"

So here is the console log as no runner is generated

[INFO] - includeScenarioTags          : @validateChangeinTaskMenuSequence-gHome and @validateSidebarTaskMenuRemoval-gHome
[INFO] - parallelizationMode          : scenarios
[INFO] - numberOfTestRuns             : 1
[INFO] -------------------------------------
[INFO] - generated   0 scenarios from src/test/resources/features/login.feature.
[INFO] - generated   0 scenarios from src/test/resources/features/pagelayout.feature.
[INFO] - generated   0 scenarios from src/test/resources/features/phaseconfig.feature.
[INFO] - generated   0 scenarios from src/test/resources/features/stepprogressbar.feature.
[INFO] - generated   0 scenarios from src/test/resources/features/enhancedflexgrid.feature.
[INFO] - generated   0 scenarios from src/test/resources/features/expressionbuilder.feature.
[INFO] - generated   0 scenarios from src/test/resources/features/forms.feature.
[INFO] - generated   0 scenarios from src/test/resources/features/approvalprocess.feature.
[INFO] - generated   0 scenarios from src/test/resources/features/sidebar.feature.
[INFO] - generated   0 scenarios from src/test/resources/features/customApp.feature.
[INFO] - generated   0 scenarios from src/test/resources/features/charts.feature.
[INFO] - generated   0 scenarios from src/test/resources/features/flexgrid.feature.
[INFO] - generated   0 scenarios from src/test/resources/features/flextable.feature.
[INFO] - generated   0 scenarios from src/test/resources/features/calender.feature.
[INFO] -------------------------------------
[INFO] Cucable created 0 separate feature files and 0 runners.

Below is failsafe config:

<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-failsafe-plugin</artifactId>
				<version>${maven.failsafe.plugin.version}</version>
				<executions>
					<execution>
						<id>Run parallel tests</id>
						<phase>integration-test</phase>
						<goals>
							<goal>integration-test</goal>
							<goal>verify</goal>
						</goals>
					</execution>
				</executions>
				<configuration>
					<forkCount>${threads}</forkCount>
					<testFailureIgnore>true</testFailureIgnore>
					<disableXmlReport>true</disableXmlReport>
					<argLine>-Dfile.encoding=UTF-8</argLine>
					<excludes>
						<exclude>**/IT*.java</exclude>
					</excludes>
				</configuration>
			</plugin>

What happens if you don't pass any tags? Are the generated runners executed?
Yes the generation looks ok. Below console log for ref:

[INFO] - generated  87 scenarios from src/test/resources/features/pagelayout.feature.
[INFO] - generated  13 scenarios from src/test/resources/features/phaseconfig.feature.
[INFO] - generated  10 scenarios from src/test/resources/features/stepprogressbar.feature.
[INFO] - generated 110 scenarios from src/test/resources/features/enhancedflexgrid.feature.
[INFO] - generated 344 scenarios from src/test/resources/features/expressionbuilder.feature.
[INFO] - generated   8 scenarios from src/test/resources/features/forms.feature.
[INFO] - generated  31 scenarios from src/test/resources/features/approvalprocess.feature.
[INFO] - generated  10 scenarios from src/test/resources/features/sidebar.feature.
[INFO] - generated   6 scenarios from src/test/resources/features/customApp.feature.
[INFO] - generated   8 scenarios from src/test/resources/features/charts.feature.
[INFO] - generated 124 scenarios from src/test/resources/features/flexgrid.feature.
[INFO] - generated  67 scenarios from src/test/resources/features/flextable.feature.
[INFO] - generated   2 scenarios from src/test/resources/features/calender.feature.

Do you actually have any scenarios that are tagged with @validLogin2 AND @validLogin3?
Yes they did. Now sharing logs with real framework using tags @validateChangeinTaskMenuSequence-gHome AND @validateSidebarTaskMenuRemoval-gHome

Answer 5 · 2019-07-17T14:52:53.000Z

In your first example, no scenarios seem to match. So naturally, there is nothing to run.

Answer 6 · 2019-07-17T14:55:24.000Z

Yes. So what is the possible fix for it ?
Shouldn't the includeScenario accept multiple tags for execution as in documentation

<includeScenarioTags>@includeMe and @includeMeAsWell</includeScenarioTags>

Answer 7 · 2019-07-17T14:56:27.000Z

Can you provide an example feature file for me to test this? Because without some data, it is hard for me to see if the problem is on your side or if it's a bug in Cucable

Answer 8 · 2019-07-17T14:58:03.000Z

here is the feature file tests

@sidebar-component @regression
Feature: Side bar Component Scenarios
  Test all sidebar component based scenarios
  As a configuration manager i make different configurational changes in side bar component and will validate them.

  @33877    @validateChangeinTaskMenuSequence-gHome
  Scenario: validate after changing sequence of task menu from second to third in side bar
    Given I have default configuration as below:
      | configurationCategory | fieldName                  | fieldOrignalValue                                                  | filterId         |
      | Side Bar Header       | Sidebar Component Sequence | Search,TaskMenu,ActivityMenu,RecentlyViewed,TechnicalSupport,Links | ApiGeneratedId:0 |
    Given I setup test configuration as below:
      | configurationCategory   | fieldName                  | fieldNewValue                                                      | coreComponentField | coreCOmponentSubFieldName | coreComponentSubFieldValue |
      | Side Bar Header:General | Sidebar Component Sequence | Search,ActivityMenu,TaskMenu,RecentlyViewed,TechnicalSupport,Links | GrantorHome        |                           |                            |
    When I login as "Automation" user
    When I setup my app as "As a Grantor"
    Then i see "Tasks" shown at number "4" in sideBar

  @33876    @validateSidebarTaskMenuRemoval-gHome @smoke
  Scenario: validate disabling of task menu from side bar
    Given I have default configuration as below:
      | configurationCategory | fieldName                  | fieldOrignalValue                                                  | filterId         |
      | Side Bar Header       | Sidebar Component Sequence | Search,TaskMenu,ActivityMenu,RecentlyViewed,TechnicalSupport,Links | ApiGeneratedId:0 |
    Given I setup test configuration as below:
      | configurationCategory   | fieldName                  | fieldNewValue                                             | coreComponentField | coreCOmponentSubFieldName | coreComponentSubFieldValue |
      | Side Bar Header:General | Sidebar Component Sequence | Search,ActivityMenu,RecentlyViewed,TechnicalSupport,Links | GrantorHome        |                           |                            |
    When I login as "Automation" user
    When I setup my app as "As a Grantor"
    Then I see "Tasks" as disabled from sidebar

Answer 9 · 2019-07-17T15:00:33.000Z

You are asking Cucable to generate scenarios that match 2 tags. But none of your scenarios match your tag expression. So Cucable will generate no scenarios.
The and keyword means: match scenarios that include both of the provided tags only.

Answer 10 · 2019-07-17T15:01:11.000Z

So if you want to match either tag, you need to use or.

Answer 11 · 2019-07-17T15:04:30.000Z

-Dtags="@validateChangeinTaskMenuSequence-gHome and @validateSidebarTaskMenuRemoval-gHome"

or

-Dtags="@33877 and @33876"

I cannot understand how these two tag names are not matching ? i have these two tags on each of the above shared tests.

Answer 12 · 2019-07-17T15:07:25.000Z

The first scenario has the tags @33877 and @validateChangeinTaskMenuSequence-gHome.
The second scenario has @33876, @validateSidebarTaskMenuRemoval-gHome and @smoke.
In your tag expression you are asking for @validateChangeinTaskMenuSequence-gHome and @validateSidebarTaskMenuRemoval-gHome.
So you have no matches.

Answer 13 · 2019-07-17T15:13:35.000Z

One test can have different tags which are independent of each other. Thats how it is always been.

Are you saying that if we have multiple tags on a test then it will be considered as consolidated 1 tag expression instead of individual tag ? It's a new thing for me though as we often have multiple tags for scenarios for grouping them during execution.

If that's so then can't we use each tag as separate tag ? isn't it a limitation ?

Answer 14 · 2019-07-17T15:15:43.000Z

oh i think what you are saying that with and we can only specify tags of the same test ? right ?

Answer 15 · 2019-07-17T15:16:37.000Z

My question is how can i run two or three tests through maven command in ?
like -Dtags=@test1,@test2,@test3

Answer 16 · 2019-07-17T15:16:51.000Z

Of course you can still use individual tags - I even told you how :-) . This is also stated in the documentation: https://github.com/trivago/cucable-plugin#includescenariotags

If you want to include scenarios that are tagged with validateChangeinTaskMenuSequence-gHome or scenarios that are tagged with validateChangeinTaskMenuSequence-gHome, you need to specify your tag expression like so
@validateChangeinTaskMenuSequence-gHome or @validateSidebarTaskMenuRemoval-gHome

The linked documentation section also has multiple examples for tag expressions and how to use them.

Answer 17 · 2019-07-17T15:18:51.000Z

Got it.
So to achieve -Dtags=@test1,@test2,@test3 i should do -Dtags=@test1 or @test2 or@test3
correct ?

Answer 18 · 2019-07-17T15:19:53.000Z

Exactly!

oh man...Thanks a lot. My bad i couldn't understand that it properly..Cheerz!!!

Answer 19 · 2019-07-17T15:20:27.000Z

No worries :-)