Let ignoring scenarios using multiple @ignore tags.
Opened this issue · 5 comments
Right now the library supports only ignoring scenarios using the @ignore
tags. However, it is a practice for other BDD libraries to have a configurable list of tags to ignore. For example, Java's Cucumber is fully configurable for each test:
tags = "not @ignore and not @ignore-custom and not @ignore-custom-something-else-whatever-you-want"
It is extremely useful for large projects. For example, when we have a number of clients for a single server, and we want to ensure that all our clients work identically. However, we want to temporarily disable some of the scenarios for specific clients, creating @ignore-client1
and @ignore-client2
.
As adding such scalable feature requires time, I'd suggest to at least give developers a way to use several different tags.
- Ideally: send a list of tags to be ignored as arguments.
- Faster: ignore all scenarios with tags
@ignore*********.
My view:
- List of tags to ignore as arguments - yes.
- Ignore all tags based on tag name pattern - no. This will create a loosely-defined, implicit convention, which I always try to avoid.
Also, in some simple cases, organizing tests using tags can help. Those can already be passed in as arguments: https://github.com/ttutisani/Xunit.Gherkin.Quick/blob/master/docs/tags.md
@ttutisani Maybe just a reversed "--filter" would be enough! Like "all except of ".
You made me think 🙂
So, good news, it's already possible because --filter
takes an expression as an argument. So, if you want to ignore all tests annotated with @ignore-xxx
tags, that means you want to run all tests that have tags that don't contain ignore. You write that like this:
filter Category!~@ignore
The only problem left is that this is not documented in the tags documentation, which is why this issue must stay open until it's documented.
Sources:
dotnet test
command-line reference explains--filter <EXPRESSION>
argument.<EXPRESSION>
part of it is explained on this page: https://learn.microsoft.com/en-us/dotnet/core/testing/selective-unit-tests?pivots=xunit#syntax
I've tested the --filter
flag. Here are my thoughts:
- It filters out tests from consideration completely, so the filtered tests are not shown in the stats of "Skipped", which is a shame;
- The runner has troubles with
!
and Bash.!
is a reserved symbol for Bash, so we need to escape it\!
, but then the runner says that something is bad with my pattern:[xUnit.net 00:00:00.6207380] Xunit.Gherkin.Quick.ProjectConsumer: Exception filtering tests: Incorrect format for TestCaseFilter Error: Invalid Condition 'Category!~@ignore'. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed.
; - It is a
dotnet
CLI flag, and it's not scalable for other builders (for example, I useBazel
, and its rules either don't use dotnet under the hood (which is a surprise) or don't pass the args to it (which is a double surprise).
So any other option would be still helpful, I guess =)