symflower/eval-dev-quality

Support multiple evaluation tasks

Opened this issue · 0 comments

As of now, we only have a single evaluation task, that is, "provide tests for this code". In a future release we want to run other tasks as well, like "repair this compilation error". For this, we need structural changes.

Tasks:

  • Introduce string enum TaskIdentifier
  • Change Model interface such that it has a RunTask(TaskIdentifier,...) method
    • Remove GenerateTestsForFile from Model interface as this becomes a task
  • Add a method IsTaskSupported(TaskIdentifier) to the Model interface to let models specify if they support a specific task.
  • Introduce proper Repository type
    • Each repository can allow/disallow certain tasks
      • This should not be hardcoded. Therefore a repository.json file can be created in the repository's root directory. If no such file exists, all tasks are allowed.
  • Create Task interface and suitable implementations
    • TaskPostRunTests which calls Model.RunTask and then runs the repository tests to validate the result