Impossible to add a benchmark that exercises newer Python features
AlexWaygood opened this issue · 1 comments
AlexWaygood commented
It appears to be currently impossible to add a benchmark that uses a feature added in Python >3.7. Even if you add the requires-python field in the pyproject.toml file for a specific benchmark, the benchmark is run on all Python versions in CI. This means that the CI will fail if the benchmark includes a feature that is new in 3.8, for example.
This issue currently blocks both of these open PRs:
brandtbucher commented
Yeah, pyperformance should probably check this somehow and refuse to run certain benchmarks based on the advertised Python version support.
Another option could be to disable them by default, and enable them when we drop older version support. But that seems less ideal.