Multiple model parameter with same value result in multiple evaluations
Munsio opened this issue · 0 comments
Munsio commented
When running the evaluation and specifying the same model multiple times currently the evaluation is run X amount of times for the model.
Example:
eval-dev-quality evaluate --runtime docker --result-path ./docker-test --runs 5 --model symflower/symbolic-execution --model symflower/symbolic-execution --model symflower/symbolic-execution --repository golang/plain
This runs the symflower/symbolic-execution
3 times with 5 runs as the model was 3 times specified as parameter.
Question:
Do we want this behavior or should we unique
the list of models after parsing?