DALPy-Developers/DALPy

Enforcing No Argument Modification in Testing Utilities

ChamiLamelas opened this issue · 1 comments

Is your feature request related to a problem? Please describe.
For problems that require students to write functions, we sometimes want to impose that the students' functions should not modify some or all of their arguments. To currently enforce this, a copy needs to be made of the arguments that aren't meant to be modified before running the students' function. Then, after the function is run, the copy needs to be compared to the arguments that were passed to the function. This occurs in cases where the arguments are data structures of some kind (e.g. Array, Stack, etc.).

Describe the solution you'd like
Since this process is easily generalizable, I believe that this should be added to the testing utilities. In particular, it could be added to the run_generic_test function.

The first question is how the user should specify which arguments should be checked for modification. This could be specified in a parameter enforce_no_mod:

  • By default, enforce_no_mod is False indicating that all of the arguments can be modified.
  • If enforce_no_mod is True, then none of the arguments can be modified and should all be checked.
  • Lastly, enforce_no_mod can be a list of bool of equal length to the existing params parameter. enforce_no_mod[i] = True means params[i] should be checked for modification. enforce_no_mod[i] = False means that it should not.

The second question is how run_generic_test will know how to copy any arguments that need to be checked. This can be specified by a list copy_fns such that copy_fns[i] is a function that can be used to make a copy of params[i]. Since params can be a single value (i.e. not a list), it may be good to allow copy_fns to be a single function also. This should be provided a default value as to not have to change existing testing code (None is sufficient).

The third question is how run_generic_test will know how to compare the argument passed to the students' function and its copy. This also can be specified by a parameter copy_eqs_fns that operates in a similar manner to copy_fns except where copy_eqs_fns[i], in the instance it's a list, is a function that checks the equality of params[i] to its copy. This should be provided a default value as to not have to change existing testing code (None is sufficient).

Describe alternatives you've considered
The alternative is the current process described in my answer to the first prompt.

Additional context

An additional reason to take this approach is that it will allow the tests display an error message describing that arguments were modified when they should not have been. This could not be easily accomplished by offloading this complexity to the user through modification to custom_comparator to include an optional params=None parameter. In such a case, the tests would simply fail despite the logged output and expected output being the same.