Apply symflower fix to a "write-test" result of a model
bauersimon opened this issue · 3 comments
bauersimon commented
Basically we want to execute the "write-test" task, but then optionally call symflower repair
on the generated tests.
Plus, the scoring should treat both the original "write-test" and the (hopefully) repaired tests as different results.
- introduce a new
TaskIdentifier
write-test-symflower-repair
that represents the write test task but with symflower applied - let
task.Run
returnmap[TaskIdentifier]Assessments
such that it can return both the unfixed, and fixed results - extend the "write-test" task such that
- if the first result fails at execution apply
symflower repair
and executes again - write the results of
symflower repair
into the original model's log, i.e.with symflower repair: .......
- watch out for the scoring... the score must be combined intelligently
- runtime probably has to be added up
- for coverage and execution only the final "attempt" matters
- ...
- if the first result fails at execution apply
-
symflower fix
needs to be merged
bauersimon commented
@Munsio plz review
Munsio commented
- When we want to "repair" something don't we need the data from the previous model? How is it shared?
- How does the
Next
Model in the chain know that it needs to repair something and not generate the thing new?
bauersimon commented
The task evaluation logic can carry over the "broken code" and just ask the next model to do a "repair". The models themselves don't need to worry about sharing context or what they need to do. The evaluation logic will say "you failed, but you, please fix, this is what we have so far".