656 total bugs or 984 total bugs?
Opened this issue · 2 comments
ahong007007 commented
When running code repair, the actual question is 656 questions,
but the paper describes
“We manually add a bug to each of the 164 HumanEval solutions across all 6 languages (984 total bugs). ”
May I ask which one is correct? The problem is also in the iterative increase?
SivilTaram commented
@ahong007007 From your log, you only select the task humanevalfixtests-python. It should contain 164 buggy questions. Would like to answer more if you have more context for the 656 number.
Muennighoff commented
Like @SivilTaram said, the Python split has 164 bugs. As there are 6 languages, there's 164 * 6 = 984 in total.
The 656 you see is the number of iterations. It is calculated as: 164 * n_samples/batch_size i.e. 164 * 4 = 656.