symflower/eval-dev-quality

Make the Knapsack.java case easier to solve for models

zimmski opened this issue · 0 comments

[`Knapsack.java`](https://github.com/symflower/eval-dev-quality/blob/main/testdata/java/light/src/main/java/com/eval/Knapsack.java) contains a nested class `Item`. Due to an oversight on our side we did not make the class `static` which means `Item` needs to be initialized with `new Knapsack().new Item()`. We were surprised to see that this caused troubles for almost all LLMs which were trying to initialize with `new Knapsack.Item()`.

Since these examples are not about language features but about how well LLMs are doing at logic and general use of a language, please fix the example so it is easier to solve.