Timeout parameter is not respected in Gradescope

[kandluis]

The timeout paramter used here is not respected at all in gradescope.

gradescope-autograde-cpp/run_autograder

Lines 82 to 83 in 90adf63

	subprocess.check_output(['./run_test'], cwd=os.path.join(gu.Config.TEST_DIR, dir_name),
	stderr=subprocess.STDOUT, timeout=settings.get('timeout', None))

.

The expected behavior would be for the process to be killed if it goes beyond the timeout. The actual behavior is that the process is left to execute until it terminates. This leads to timeout-based tests failing causing the entire test suite to timeout.

This is due to a bug in the way the Python handles the timeout paramter when multiple subprocesses are spawned. See https://bugs.python.org/issue37424.

This was fixed in Python 3.7.5, however, Gradescope base images still run with Python 3.6.9 which does not include the fix.

A workaround is to use the following check_output function instead:

def check_output(*popenargs, timeout=100, **kwargs):
    """Our own implementation of open that actually kills with the timeout.

    See # See: https://stackoverflow.com/questions/36952245/subprocess-timeout-failure.
    """
    with subprocess.Popen(
        *popenargs, 
        stdout=subprocess.PIPE,
        preexec_fn=os.setsid, **kwargs) as process:
        try:
            output, _ = process.communicate(timeout=timeout)
        except subprocess.TimeoutExpired:
            os.killpg(process.pid, signal.SIGINT) # send signal to the process group
            output, _ = process.communicate()
            raise subprocess.TimeoutExpired(process.args, timeout, output=output)

The above basically re-implents check_output to use os.killpg which kills the entire process group (avoiding any issues with subprocesses still running and having access to the pipe, thereby forcing the program to block).

For Gradescope, the solution would be to upgrade from Python 3.6.9 to >= 3.7.5 -- however, if that's not feasible, I recommend using the above instead otherwise the timeout paramter is completely ignored.