mila-iqia/milabench

Validation Layers

Delaunay opened this issue · 0 comments

Benchmarks are hard to tests but we can implement verification through giving & voir

  • Validate that the right number of jobs are launched
  • Validate that the right accelerator is being used
  • Validate that the values are not Nan
  • Validate that accuracy is going down

The validation could be enabled/disabled to avoid taxing the underlying task we are evaluating,
they will always be enabled for the CI