SanityCheck: take inspiration from what `verifyproblem` does (kattis tools)
Opened this issue · 1 comments
wil93 commented
The verifyproblem
tool (see https://github.com/Kattis/problemtools) has many interesting sanity checks that we most likely would benefit from, for example:
- it warns if the input / output files don't have a newline at the end of the file
- it performs some fuzzying of the validators, it checks that at least one validator rejects inputs with:
- spaces added where there already is whitespace
- newlines added where there already are newlines
- leading zeros added to integers
- trailing zeros added to real number decimal portion
- random junk added to the end of the file
And I guess other checks that I don't remember or that I didn't run into
edomora97 commented
I've skimmed through the verifyproblem source code and listed the interesting checks:
- Check input/output newline format (
\n
vs\r\n
) - Check attachment input/output newline format (
\n
vs\r\n
) - Check input files end with
\n
- Check attached files end with
\n
- Identical input files in a subtask
- No directory in attachments
- Fuzz custom checker with junk files (should reject with score 0 and no crash):
a. an empty file
b. a binary file with byte values 0 up to 256
c. a text file with the ASCII characters 32 up to 127
d. a random text file with printable ASCII characters - Fuzz custom checker with junk modifications (when no grader: should not change the score):
a. spaces added where there already is whitespace
b. newlines added where there already are newlines
c. leading zeros added to integers
d. trailing zeros added to real number decimal portion
e. random junk added to the end of the file
Most of them make sense, others may require some discussions (e.g. (1) what about with binary I/O? (5) may be useful? I guess no).
All (or most of them at least) should be quite easy to implement.