symflower/eval-dev-quality

Multiple model parameter with same value result in multiple evaluations

Opened this issue · 0 comments

Munsio commented

When running the evaluation and specifying the same model multiple times currently the evaluation is run X amount of times for the model.

Example:

eval-dev-quality evaluate --runtime docker --result-path ./docker-test --runs 5 --model symflower/symbolic-execution --model symflower/symbolic-execution --model symflower/symbolic-execution --repository golang/plain

This runs the symflower/symbolic-execution 3 times with 5 runs as the model was 3 times specified as parameter.

Question:
Do we want this behavior or should we unique the list of models after parsing?