Bug with adding a new dataset, failing evaluation if name is misformatted / round is not selected
ktirumalafb opened this issue · 4 comments
I found a bug when trying to modify eval server, where:
(1) if the name of an uploaded dataset is not [a-ZA-Z] (for example squad_v2
)
OR
(2) a round is not selected (the default round number for a dataset is 0)
then batch transform jobs are sent through and the model is evaluated, but the update to the DB fails. So essentially, we do all the work of evaluation, but it still shows up as "failed" and the job metrics are lost. We should either (a) force task owners to correctly format the dataset name + select a round when they upload a dataset, before running evaluation or (b) have a sensible default way of handling this (for example, getting rid of unwanted characters in dataset name to solve (1) + selecting the latest round to solve (2))
note: the round 0 portion (number (2) above) might not be true in prod (i.e. prod db might have some default round 0 entry in the rounds table), but it does cause a local eval server to not update the DB with metric and fail instead
Oh nice catch! I just created an issue for this but didn't see that you beat me to it.
Specifically, I think that the dataset name needs to match this regex: ^[a-zA-Z0-9]{0,62}
btw, should we still force a valid round to be selected on dataset upload?