symflower/eval-dev-quality

Rethink retry logic for LLM Providers

Munsio opened this issue · 0 comments

This image shows the uptime of the RWKV v5 World 3B model:
image

This model is so long down that the retry logic we are currently using does not really work, but waiting for the model to be available again would only stretch out the evaluation runs artificially