Group scenarios whose names match REGEXP together
Closed this issue · 21 comments
When parallelizationMode = scenarios, group scenarios whose names match REGEXP together, in the same runner file if possible (inspired by cucumber.options="--name REGEXP").
Use Case:
I have a scenario where different types of users need to be tested for different capabilities (capabilities can be anything, for example, presence or absence of UI elements depending on the logged in user). Login is a very expensive operation which involves multiple steps and can take up to 30 seconds. So I would like to minimize the number of logins as much as possible. The scenario outline looks like this:
Scenario Outline: Test "<capability>" of "<user>"
When I log in as a "<user>"
Then I can test "<capability>"
Scenarios:
| user | capability |
| user1 | cap1 |
| user1 | cap2 |
| user1 | cap3 |
| user2 | cap4 |
| user2 | cap5 |
| user3 | cap6 |
| user3 | cap7 |
With desiredNumberOfRunners set to 3, what I see now is:
runnerFile1:
Test "cap1" of "user1"
Test "cap4" of "user2"
Test "cap7" of "user3"
runnerFile2:
Test "cap2" of "user1"
Test "cap5" of "user2"
runnerFile3:
Test "cap3" of "user1"
Test "cap6" of "user3"
This will lead to multiple logins in each runner.
What I would like to do is to be able to specify a REGEXP , in this case user\d, and see the scenarios grouped as below:
runnerFile1:
Test "cap1" of "user1"
Test "cap2" of "user1"
Test "cap3" of "user1"
runnerFile2:
Test "cap4" of "user2"
Test "cap5" of "user2"
runnerFile3:
Test "cap6" of "user3"
Test "cap7" of "user3"
If this seems to be something worth doing, I would also like to propose an extension of the above where the user can specify a list of REGEXP, say regex1, regex2, regex3, and desiredNumberOfRunners automatically gets set to the number of items in the list (in this case, 3). The scenarios matching each REGEXP can then be moved to separate runner files.
Interesting idea although to me it seems like a very special case you have there. Would you be able to prepare a pull request for this?
Special case indeed! 😄
I am pretty much swamped with developing tests for the same app described above (7x combinations for each test). But I will see if I can find some time over the coming weekends.
@laxersaz, I have been thinking about this and here is a rough outline of an idea using custom template placeholders, [CUCABLE:CUSTOM:xxx] and name option in @CucumberOptions.
Define the REGEXPs as custom values. Using the example above:
<customPlaceholders>
<name1>user1</name1>
<name2>user2</name2>
<name3>user3</name3>
</customPlaceholders>
Now generate 3 (count equal to the number of strings defined) runner files as below, including name in @CucumberOptions.
@CucumberOptions(
features = {"target/parallel/features"},
plugin = {"json:target/cucumber-report/[CUCABLE:RUNNER].json"},
name = [CUCABLE:CUSTOM:name1/name2/name3]
)
public class CucableJavaTemplate {
}
The split feature files do not have to be added to features = {} as name will decide the scenarios that will be executed by this runner.
I haven't looked at the code yet but wanted to know your thoughts. Do you think this is a practical approach?
It is an interesting approach indeed. I cannot really judge if this works as intended without trying though.
I just tried a simple test by adding name = string1 to the runner template and executing 4 tests with desiredNumberOfRunners set to 2. string1 matches only a single test out of 4. 2 runners were generated, one of them executed 0 tests and the other executed only the test matching string1. So it seems to work as expected.
Can this be closed then?
No @laxersaz. The runner files still have to be generated and the strings added to them, 1 string per runner. Also, [CUCABLE:FEATURE].feature has to be trimmed from features = {}.
If I have this list:
<customPlaceholders>
<name1>string1</name1>
<name2>string2</name2>
<name3>string3</name3>
</customPlaceholders>
3 runners have to be generated as below:
@CucumberOptions(
features = {"target/parallel/features"},
plugin = {"json:target/cucumber-report/[CUCABLE:RUNNER].json"},
name = "string1"
)
public class CucableJavaTemplate {
}
@CucumberOptions(
features = {"target/parallel/features"},
plugin = {"json:target/cucumber-report/[CUCABLE:RUNNER].json"},
name = "string2"
)
public class CucableJavaTemplate {
}
@CucumberOptions(
features = {"target/parallel/features"},
plugin = {"json:target/cucumber-report/[CUCABLE:RUNNER].json"},
name = "string3"
)
public class CucableJavaTemplate {
}
@laxersaz, I have started working on this.
@laxersaz, I implemented this. But during my testing, I found that this will work as expected only when no cucumber options are passed from the command line (-Dcucumber.options). cucumber.options passed from the command line will completely override those mentioned in the runner class (per this thread). I would still like to have the ability to pass command line options. So I am looking at adding only the matching scenarios to the runner class as discussed in original description.
@laxersaz, I have completed the work on this one. Please review the PR you get a chance.
It seems like there is still a bug in the implementation. When I use "Scenario 1, Scenario 2", it will also include "Scenario 3".
@laxersaz, thanks for checking. I will take a look.
Was Scenario 1, Scenario 2 the specific values that failed, or was that an example? Can you also send me the lines starting with Scenario: and the scenario descriptions (if any) in the split feature files please?
I used this in the Cucable example project. You can try it out with the current one from master.
@laxersaz, I executed the example project with scenarioNames specified as Scenario 1, Scenario 2.
2 runners were generated, one with 6 feature files matching Scenario 1 and the other with 2 feature files matching Scenario 2. I do not see Scenario 3 included.
[INFO] -------------------------------------
[INFO] Cucable Maven Plugin, version 1.5.3
[INFO] -------------------------------------
[INFO] Cucable created 30 separate feature files and 2 runners.
Runner 1 - Scenario 1
features = {"target/parallel/features/My_feature_dashes_scenario001_run001_IT.feature",
"target/parallel/features/MyTest1_scenario001_run001_IT.feature",
"target/parallel/features/MyTest2_scenario001_run001_IT.feature",
"target/parallel/features/MyTest1_scenario006_run001_IT.feature",
"target/parallel/features/MyTestSub_scenario001_run001_IT.feature",
"target/parallel/features/MyTest1InTestFeature2_scenario001_run001_IT.feature"}
Runner 2 - Scenario 2
features = {"target/parallel/features/MyTest1_scenario002_run001_IT.feature",
"target/parallel/features/MyTest2_scenario002_run001_IT.feature"}
Can you please let me know the steps you performed when you saw the issue?
You are right. However, I was thrown off by the Cucable created 30 separate feature files and 2 runners.
Shouldn't it only generate the features that match?
Currently, the matching of scenarioNames happens after all the individual feature files are generated. I thought this will be the way to support a Scenario Outline: when the user specifies a value in the Scenarios: table in the scenario name (as seen in the original description).
Do you think the matching should happen at the gherkin parsing stage itself? I can look into that.
I think this would be much less confusing for users to have just the matching feature files generated 😏
You can take a look at the scenarioShouldBeIncluded() method in the GherkinDocumentParser. This does a similar thing for tags :-)
Thanks @laxersaz. I will try to work on this over the weekend (if not earlier).
This is mostly done. I will add tests and create a PR tomorrow.