symflower/eval-dev-quality

Fixed timeouts for `symflower unit-tests` and `symflower test`

Opened this issue · 2 comments

During the run of the light repository with the ollama models I encountered a problem where symfower test was running for ~1H (stuck). In order to avoid this we need to set sane timeouts for all those execution calls.

Notes:

  • Maven downloads may take also a while, should we add a "prefetch" state for dependencies so we can retry in the case of an error. (We know that maven is sometimes flaky)
  • I think a sane execution time limit for running tests would be 5-10 minutes. If we are going for a prefetch stage than it should be on the lower end.
  • We should probably also use the timeout error as metric.

@bauersimon @ahumenberger @ruiAzevedo19 I've added myself here as assignee but everyone can take this Issue.
I also believe we should fix this sooner then later as it is currently blocking the execution of v0.5.0

language/java/language.go:96
language/golang/language.go:100
model/symflower/symflower.go:42

I guess those are the parts where we need to apply the timeout.