Use native string as test case name

[mjirous]

I would like to propose an enhancement that would bring another way to define a test case. Currently there is only TEST, TEST_F etc. It has the limitation that the test case name must conform c++ function name restrictions and therefore only a limited set of characters can be used.

This enhancement would introduce a new set of macros (e.g. SCENARIO, SCENARIO_F ...) for test definition that would allow native c string as test case name. With this change we could use basically all characters which would lead to test case names consisting from spaces, special characters like brackets and so on.

Consider the following simple example:
TEST_F(MyFixture, When_customer_orders_beer_the_item_is_added_to_current_order)

SCENARIO_F(MyFixture, "When customer orders beer the item is added to current order")

The reason why to incorporate such change to gtest is to simplify test description. People tend to write test cases names that are too short, because it is not convenient to describe the test without spaces. The consequence is that tests are hard to understand, because the description does not explain test purpose. But when the name could be described using a normal string with normal language and all characters, then it would be much easier to describe the test better in normal sentences. And it also looks and reads better.

Besides Catch already supports this so why not having this in gtest?

The proposed SCENARIO_F would support normal string instead of just function name. Implementation has already been done in the CppBDD project and is working just fine. The implementation is no magic, because gtest handles the test case name as string anyway, so only the new macro has been introduced. We have been using this for quite some time, but we think that it would be great to have this feature in gtest.

There is only one thing to consider. The parameterized tests are formatted with special character (#) in the test list. Some test adapters used in IDEs use this and try to parse the content. So maybe at least this character would have to be replaced to avoid test case name parse issue.

I would like to know whether this makes sense. Thanks.

[gennadiycivil]

Thank you for this report. We are actively discouraging adding new macros

[devans80]

@gennadiycivil can you explain why, or outline an alternative way to achieve the same result in a way that's more acceptable?

[gennadiycivil]

@devans80
Two parts to the answer

1. Using native strings to test case names. We are open to hear about this idea - however there would need to be a clear and compelling argument why the "new thing" is "better" and what it would allow is to do that is not currently possible.
   Why is using native strings in test case naemes advantageous?

2. Macros. We state that the existing API surface with macros it already sufficient for vast majority of uses. If there is sufficient proof that new macros would allow us to do something that is not already possible (or there is a large demonstrable improvement) we could re-consider. However the bar is high on this.

Thanks again

[devans80]

@gennadiycivil
Thanks for your reply. These are fair questions.
The argument in favour of expressing tests with natural looking language (allowing for spaces), is the same argument put forward by proponents of Behaviour Driven Development (BDD). There are many articles that can do a better job of justifying this than I, so here is one example:
http://www.its-all-design.com/behaviour-driven-development-use-cases-re-invented/
So if you're not convinced that BDD is useful, it follows that adding spaces is not useful either. So this addresses the 'is it better' part of the question. As for it being a 'new thing' well, BDD has been around for a while, just not in googletest.
I agree with the point about Macros, and will have a think about how the same result might be achieved without them.

[mjirous]

@gennadiycivil @devans80 There is a way how to achieve the same results without using a new marco. The TEST and TEST_F could be used for native string test case name as well. Basically, the test case name marco argument could accept the currently supported name but the string in quotes as well. The # preprocessor operator works for both.
However, it is not for free. TEST and TEST_F uses the test case name for class name, which would obviously won't be possible with name in quotes. The class name would have to be auto generated the way the gtestbdd does it currently for SCENARIO and SCENARIO_F (part of the name is line number). This may have some unforeseen consequences. It may break compatibility with other tools integrating with gtest like Google Test Adapter for Visual Studio. On the other hand Resharper c++ and CLion integrates well with SCENARIO.
Therefore, I would prefer to rather introduce a new macro, but I don't mind changing TEST, TEST_F and others.

[devans80]

@mjirous

Therefore, I would prefer to rather introduce a new macro, but I don't mind changing TEST, TEST_F and others.

Did you mean "I would prefer not to introduce a new macro" ? 🙂
It sounds like modifying TEST and TEST_F is promising, but can you clarify if you're suggesting there might be an alternative way to auto-generate class names that doesn't break Google Test Adapter or Visual Studio?

[mjirous]

@devans80 Due to backward compatibility the new macro would be better. If it was just up to me I would change TEST_F, because it will not cause any issue to our team, but there are other developers around the world that may depend on the current implementation although the class name is not part of public interface. This is up to the gtest devs.
I don't see a way to avoid breaking GTA, because it basically takes pdb file and tries to map class name to test case name, which is not possible once the class name is autogenerated. Resharper and CLion expands the macro and finds the right name.

[gennadiycivil]

Good discussion, thank you. The best way to approach this would be to create a proper PR and submit it for consideration.

[tapika]

Please note - this might coecho with Google test adapter requirement - "Add support for data driven tests", see following link:

csoltenborn/GoogleTestAdapter#102

and like I've mentioned you need only to recombine right pieces of code to right places.

Please let me know if there is anything I can help you with.

[asoffer]

Thanks for bringing this up. We are sympathetic to the request and do generally like the approach, but this is not a direction we intend to pursue for the reasons Gennadiy outlined above.