nuprl/MultiPL-E

Java program evaluation error with javatuples.Pair class

PootieT opened this issue · 2 comments

PootieT commented

Hi,

I am evaluating on Java completions, and seems like whenever the completion uses javatuples.Pair program, the evaluation throws the following error

Error: Unable to initialize main class Problem
Caused by: java.lang.NoClassDefFoundError: org/javatuples/Pair

I tried two ways of evaluating, using local environment, downloaded javatuples-1.2.jar file and set CLASSPATH in env var in evaluation/src/eval_java.py, as well as trying the existing podman container ghcr.io/nuprl/multipl-e-evaluation. Both yields the same result for 9 out of 158 programs in HumanEval. Is this supposed to happen? seems like some evaluation script fault?

arjunguha commented

Thanks for reporting this. I believe I've fixed the issue in the Java evaluation script and uploaded a new container as well. Would you try again?

FYI @abhijangda

PootieT commented

Thanks for the quick fix! I implemented your fix locally and it worked. I assume the new container will work as well.