Update sorting in integration tests for outputs from the same fold scope
chewselene opened this issue · 0 comments
In our integration tests (tests/integration_tests/test_backend_integration.py
) we sort the output for comparison. For list outputs, each list is sorted independently. However, folded outputs from the same fold scope cannot be sorted independently. For example, if english_numbers
and spanish_numbers
come from the same fold scope we may receive results like:
[{"english_numbers": ["one", "two", "three"], "spanish_numbers": ["uno", "dos", "tres"]}]
but, because we sort the list output individually, the test would check:
[{"english_numbers": ["one", "three", "two"], "spanish_numbers": ["dos", "tres", "uno"]}]
We don't currently have a straightforward way to tell whether or not outputs came from the same fold scope so this will require figuring out a good way to find that info.