NVIDIA/nvbench

Add benchmark groups/sets/tags

pauleonix opened this issue · 3 comments

It would enable a nicer (shorter) way of specifying benchmarks in the CLI interface if one could define groups/sets of benchmarks in the source code.

One way of doings this could be to leverage existing namespaces, i.e. being able to call bench --namespace foo to execute all benchmarks in that namespace (and sub-namespaces). Although I don't know how easy this is to implement compared to just adding a new class.

Using namespaces would also be somewhat limiting as one can't have two namespaces with partial overlap.

What comes to mind is how Catch2 uses tags.

We could adapt this idea to nvbench to look something like:

NVBENCH_BENCH(benchmark).add_tags("[foo][bar]")

You could then specify on the CLI one or more tag strings and it would run only the benchmarks with that tag.

I like the idea, and the suggestion to use familiar Catch2 syntax is definitely a good direction to take. I'd been planning to add regex matching to the -b <benchmark string>, but this would provide greater flexibility 👍