/FixEval

We introduce FixEval , a dataset for competitive programming bug fixing along with a comprehensive test suite and show the necessity of execution based evaluation compared to suboptimal match based evaluation metrics like BLEU, CodeBLEU, Syntax Match, Exact Match etc.

Primary LanguagePythonMIT LicenseMIT

Stargazers

No one’s star this repository yet.