microsoft/adaptive-testing

Create a flexible template-based test structure

Closed this issue · 1 comments

Right now all tests are constrained to have the form Prefix "value1" comparator "value2". While this supports the current set of tests, it is a bit obtuse (odd that the changing comparator changes the meaning of the values) and does not support any test types with more than two values. We don't want to change this after release since it would cause compatibility issues. Here is a proposal for a new system:

Allow for a free form test format with placeholders for values and outputs. Such as

  1. {} should not output {}
  2. {} should output {}
  3. {} should have the same output as {}
  4. {} should be more {} than {}
  5. {} should match {} more than {}
  6. etc.

These can all be called "test_format" instead of "comparator". Scorers will need to support each test format specifically, but need not support all of them if they are not used. The columns value1, value2, etc. match the {} placeholders in order.

Closing this issue since now AdaTest uses local topic models that implicitly define an expectation (instead of having a set of explicit hard coded expectation types).