Add optional types to type axes

[pauleonix]

I would be nice to be able to specify

NVBENCH_BENCH_TYPES(my_benchmark, NVBENCH_TYPE_AXES_OPT({my_types, my_optional_types}))
  .set_type_axis_names({"ValueType"});

such that my_bench_executable --benchmark my_benchmark does only execute (the Cartesian product of) the non-optional my_types, but my_bench_executable --benchmark my_benchmark -a ValueType=SomeOptionalType still works, i.e. the full Cartesian product of the concatenation of my_types and my_optional_types with other type axes (not included in above code) is instantiated.

[pauleonix]

Is there some kind of design decision/philosophy that running the benchmark executable without any arguments should always run all possible configurations? If not I would file another similar issue for adding benchmarks that have to be specified explicitly on the command line to be executed.

[jrhemstad]

Using Catch2 for inspiration again, they have a notion of "hidden" tests that will be skipped by default, but can be executed by explicitly specifying the test name or the "hidden" tag.

[alliepiper]

I think the hidden tag approach might not be sufficient here, since AIUI the tags would be applied at the benchmark level, not at the configuration level.

I suppose it could still work if the hidden / default type axes were split into two benchmarks, e.g.

NVBENCH_BENCH_TYPES(my_benchmark, NVBENCH_TYPE_AXES(my_types));
NVBENCH_BENCH_TYPES(my_benchmark_extra, NVBENCH_TYPE_AXES(my_optional_types)).add_tags({"hidden"});

For a long-term solution, I'm not a fan of adding this via a macro, since macros are difficult to extend/maintain and we'll run into combinatorial issues as we add more features to them. We might be able to add some API to benchmark_base that would handle this within the same benchmark. This may be a good motivation to add something similar to GBench's Apply(...) that allows a functor/lambda to perform more programmatic modification of benchmark specs.

[alliepiper]

Maybe Something like

benchmark->hide_configuration<TypeParam1, TypeParam2, ...>(value_param_1, value_param_2, ...);

or even

benchmark->tag_configuration<TypeParam1, TypeParam2, ...>({"hidden"}, value_param_1, value_param_2, ...);