Simply grade student assignments made in Java or anything that runs on the JVM (Scala/Kotlin/Jython/...).
@Test
@Grade(value = 5, cpuTimeout=1000)
@GradeFeedback("Are you sure your code is in O(n) ?", onTimeout=true)
@GradeFeedback("Sorry, something is wrong with your algorithm", onFail=true)
void yourtest() {
//a test for the student's code
}Features:
- CPU timeouts on the code
- Jails student code
- No I/O, including stdout/err
- No thread creating by the student, ...
- Most things involving syscalls are forbidden
- specific permissions can be added on specifics tests if needed
- Text/RST reporting
- Custom feedback, both from outside the test (onFail, onTimeout, ...) but also from inside (see below).
We use this library at UCLouvain in the following courses:
- Data Structures and Algorithms (LSINF1121)
- Computer Science 2 (LEPL1402)
- Constraint Programming (LING2365)
This library is best used with an autograder, such as INGInious.
Add the @Grade annotation on your JUnit test like this:
@RunWith(GradingRunner.class)
public class MyTests {
@Test
@Grade(value = 5)
void mytest1() {
//this works
something();
}
@Test
@Grade(value = 3)
@GradeFeedback("You forgot to consider this particular case [...]", onFail=true)
void mytest2() {
//this doesn't
somethingelse();
}
}Note that we demonstrate here the usage of the @GradeFeedback annotation, that allows to give feedback to the students.
You can then run the tests using this small boilerplate:
public class RunTests {
public static void main(String args[]) {
JUnitCore runner = new JUnitCore();
runner.addListener(new GradingListener(false));
runner.run(MyTests.class);
}
}This will print the following on the standard output:
--- GRADE ---
- class MyTests 8/8
mytest1(StdTests) SUCCESS 5/5
ignored(StdTests) FAILED 0/3
You forgot to consider this particular case [...]
TOTAL 5/8
TOTAL WITHOUT IGNORED 5/8
--- END GRADE ---
Everything needed is located inside the files:
- Grade.java: annotation for grading a test
- GradeFeedback.java: annotation to add feedback when a test fails/succeeds/timeouts/is ignored
- GradeClass.java: annotation to grade all tests from a class, and to give an overall score
- CustomGradingResult.java: exception to be thrown by the "teacher" to give custom score/feedback, if needed
To add it as a dependency of your project, you can add this to your pom.xml in maven:
<dependency>
<groupId>com.github.guillaumederval</groupId>
<artifactId>JavaGrading</artifactId>
<version>0.5.1</version>
</dependency>If you are not using maven, search.maven probably has the line of code you need.
It is (strongly) advised when using an autograder (did I already say that INGInious is a very nice one?) to put a maximum time to run a test:
@Test
@Grade(value = 5, cpuTimeout=1000)
void yourtest() {
//a test for the student's code
}If the test runs for more than 1000 milliseconds, it will receive a TIMEOUT error and receive a grade of 0/5.
Note that if you allow the student (via the addition of some permission) to create new threads, the time taken in the new threads won't be taken into account!
It is also possible to add a wall-clock-time timeout, via JUnit:
@Test(timeout=3000) //kills the test after 3000ms in real, wall-clock time
@Grade(value = 5)
void yourtest() {
//a test for the student's code
}By default, setting a CPU timeout also sets a wall-clock timeout at three times the cpu timeout.
If you want to override that, set a different value to @Test(timeout=XXX).
Ignored tests are supported:
@Test
@Grade(value = 5)
void yourtest() {
Assume.assumeFalse(true); //JUnit function to indicate that the test should be ignored
}Use the @GradeFeedback annotation to give feedback about specific type of errors
@Test
@Grade(value = 5)
@GradeFeedback("Congrats!", onSuccess=True)
@GradeFeedback("Something is wrong", onFail=True)
@GradeFeedback("Too slow!", onTimeout=True)
@GradeFeedback("We chose to ignore this test", onIgnore=True)
void yourtest() {
//
}Throw the exception CustomGradingResult to give a custom grading from inside the text.
In order to avoid that students throw this exception, this feature is disabled by default. You must activate it by
setting @Grade(custom=true) and protect yourself your code against evil students that may throw the exception themselves.
@Test
@Grade(value = 2, cpuTimeout=1000, custom=true)
void yourtest() {
try {
//code of the student here
}
catch (CustomGradingResult e) {
throw new CustomGradingResult(TestStatus.FAILED, 0, "Well tried, but we are protected against that");
}
if(something) {
throw new CustomGradingResult(TestStatus.FAILED, 1, "Sadly, you are not *completely* right.");
}
else if(somethingelse) {
throw new CustomGradingResult(TestStatus.FAILED, 1.5, "Still not there!");
}
else if(somethingentirelydifferent) {
throw new CustomGradingResult(TestStatus.TIMEOUT, 1.75, "A bit too slow, I'm afraid");
}
else if(otherthing) {
throw new CustomGradingResult(TestStatus.SUCCESS, 2.5, "Good! Take these 0.5 bonus points with you");
}
//by default, if you throw nothing, it's SUCCESS with the maximum grade
}When using an autograder (I may already have told you that INGInious is very nice) you might want to output something nice (i.e. not text) for the students. JavaGrading can output a nice RestructuredText table:
public class RunTests {
public static void main(String args[]) {
JUnitCore runner = new JUnitCore();
runner.addListener(new GradingListener(true)); //notice the *true* here
runner.run(MyTests.class);
}
}
The @GradeClass annotation allows setting a default grade for all test (avoiding to put @Grade everywhere)
and also to give an overall max grade for the whole class. See next example for... an example.
JUnit's parameterized tests are also supported:
import com.github.guillaumederval.javagrading.Grade;
import com.github.guillaumederval.javagrading.GradeClass;
import com.github.guillaumederval.javagrading.GradingRunnerWithParametersFactory;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import java.util.Arrays;
import java.util.Collection;
@RunWith(Parameterized.class)
@Parameterized.UseParametersRunnerFactory(GradingRunnerWithParametersFactory.class)
@GradeClass(totalValue = 100)
public class ParametersTests {
@Parameterized.Parameters
public static Collection numbers() {
return Arrays.asList(new Object[][] {
{ 1 },
{ 2 },
{ 3 },
{ 4 },
{ 5 }
});
}
int param;
public ParametersTests(int param) {
this.param = param;
}
@Test
@Grade(value = 1)
public void mytest() throws Exception {
if(param % 2 != 0)
throw new Exception("not even");
}
}Output:
- class ParametersTests 40/100
mytest[0](ParametersTests) FAILED 0/20
mytest[1](ParametersTests) SUCCESS 20/20
mytest[2](ParametersTests) FAILED 0/20
mytest[3](ParametersTests) SUCCESS 20/20
mytest[4](ParametersTests) FAILED 0/20
If you have multiple test classes, simply update the main function like this:
public class RunTests {
public static void main(String args[]) {
JUnitCore runner = new JUnitCore();
runner.addListener(new GradingListener(false));
runner.run(MyTests.class, MyTests2.class, MyOtherTests.class /*, ... */);
}
}JavaGrading installs a custom SecurityManager that forbids the tested code to do anything that it should not do.
It effectively forbids a lot of things.
JavaGrading adds an additionnal permission to this list, namely PrintPermission, that allows the test code to
print things on stdout/stderr.
You can re-enable some permissions for a specific test if needed, but it does requires some boilerplate:
@RunWith(GradingRunner.class)
public class PermissionTest {
@Test
@Grade(value = 5.0, customPermissions = MyPerms1.class)
public void allowPrint() {
System.out.println("I was allowed to print!");
}
@Test
@Grade(value = 5.0, customPermissions = MyPerms2.class)
public void allowThread() {
Thread t = new Thread() {
@Override
public void run() {
// nothing
}
};
t.start();
}
/*
NOTE: the class MUST be public AND static (if it is an inner class) for this to work.
=> it must have an accessible constructor without args.
*/
public static class MyPerms1 implements Grade.PermissionCollectionFactory {
@Override
public PermissionCollection get() {
Permissions perms = new Permissions();
perms.add(PrintPermission.instance);
return perms;
}
}
public static class MyPerms2 implements Grade.PermissionCollectionFactory {
@Override
public PermissionCollection get() {
Permissions perms = new Permissions();
perms.add(new RuntimePermission("modifyThreadGroup"));
perms.add(new RuntimePermission(("modifyThread")));
return perms;
}
}
}