It seems that models that fail are not taken down anymore?
TristanThrush opened this issue · 4 comments
@TristanThrush Do you mean via prune_models.py (the model pruning scripts we added a while back)? Because I don't think any of those jobs were run as cron jobs (they just ran as one time things)
No, I mean models that fail during evaluation on a dataset are not automatically taken down. But, actually I'm not sure if we want to do this anymore anyway.
A related issue now is that, when someone uploads a dataset and a model fails, the evaluation server will assume that there is a problem with that model and not ever run an evaluation with that model again. But there could be a problem with that dataset instead, so we should enable re-evaluation to happen even when a model failed in the past. I can take this on.
#865 should have fixed