oughtinc/ice

Skipping slow tests

Opened this issue · 1 comments

When I run ./scripts/run-tests.sh, all tests show PASSED except two:

tests/test_metrics.py::test_nubia FAILED
tests/test_metrics.py::test_gold_paragraphs SKIPPED (Parses all PDFs - very slow)
  1. test_nubia fails with an obscure pydantic validation error, I had to add a print to understand the problem. How about adding @mark.skipif(not settings.OUGHT_INFERENCE_API_KEY, reason='...')?
  2. Both tests have @mark.slow but this doesn't do anything by default. I assume this is why test_gold_paragraphs has a hard @mark.skip. This doesn't seem great - what do you do when you actually want to run the test? Should a way to skip slow tests by default (with or without using @mark.slow) be investigated?
  3. -m 'not slow' correctly deselects the slow tests, but only if I directly run docker compose and pytest. ./scripts/run-tests.sh -m 'not slow' doesn't work, and the shell (tested both bash and zsh) shows it running with -m 'not\' slow. Looks like this isn't working:

# https://superuser.com/questions/403263/how-to-pass-bash-script-arguments-to-a-subshell
extra_pytest_args="$(printf "${1+ %q}" "$@")" # Note: this will have a leading space before the first arg

This definitely seems worth fixing as part of an overhaul of all the scripts: #7 (comment)