Improve test case selection mechanism

[brettlangdon]

Right now CLI accepts a regex pattern of what to run, but it only allows you to filter on the suite name.

Instead we should allow people to:

• Run all suites
  • Default behavior?
• Run specific suites
  • e.g. riot run black flake test
• Run suites based on a matching pattern
  • e.g. "^django*" (to match "django", "djangorestframework", etc)
• Run a specific case
  • Run a suite with a specific set of dependencies selected, like Python 3.8 and Django 2.0.0
• Run cases based on a matching pattern
  • Either scoped to a specific suite, or across all suites
  • e.g. run all Python 3.8 tests across all suites

We should try to enumerate all the cases we have that we need to run before we try to design a CLI/API interface for it.

[brettlangdon]

We might want to limit/select on:

• Suite(s)
• Python version(s)
• Case(s)
• Specific dependency version(s)

Either by a specific exact match, or regex pattern ?

[Kyle-Verhoog]

I wonder if it makes sense to offer what we end up doing with tox by just smush all the various pieces together into a string and then just accept a regex pattern.

eg. "python2.7-django1.3-...." which can be limited/selected on with a regex. Managing between versions of a library might be tricky if that's something we want to offer.