Benchmark Groups Have Some Minor Problems

Question

Benchmark Groups Have Some Minor Problems

ericsnowcurrently opened this issue 2 years ago · 3 comments

ericsnowcurrently commented 2 years ago

(See pyston/python-macrobenchmarks#10.)

a benchmark group "all" is allowed, but ignored
duplicate groups are allowed (between manifest files) but the semantics aren't clear
a group will overwrite a tag with the same name, without warning
the "list_groups" command does not show tags if any custom groups are defined
tags aren't found properly in BenchmarksManifest.resolve_group()

Answer 1 · 2022-08-19T22:15:27.000Z

Also "list -b TAG" isn't working.

Answer 2 · 2022-08-19T22:18:31.000Z

Also, if the "default" group includes a name that does not exist then it is ignored, whether or not "-NAME". I'd expect it to fail like it does for any other group.

Answer 3 · 2022-08-19T22:22:16.000Z

@kmod, the main problem from pyston/python-macrobenchmarks#10 is due to how a custom group named "all" is getting ignored. I'm probably going to just disallow such a group.