Uncouple tests from core
ehmatthes opened this issue · 7 comments
ehmatthes commented
Right now, integration and e2e tests are tightly coupled with django-simple-deploy core. Uncouple these tests, so all platform-specific integration and e2e tests are within the plugin directories.
- Pull
build_dev_env.py
into developer_resources. Consider developer_tools, distinct from "resources". - Uncouple platform-specific integration tests.
- Uncouple platform-specific e2e tests.
- Run black.
- Test on Windows.
- Merge.
- Update changelog.
ehmatthes commented
Integration tests
- Make sure there's no tight coupling in:
- Root conftest.py. Coupled through "tests/e2e_tests".
- integration_tests/conftest.py
-
__init__.py
-
utils/it_helper_functions.py
Coupled through path to reference files. -
utils/manage_sample_projects.py
. Coupled throughfiles_dirs_to_remove
.
- Can I just move a platform's tests to its plugin dir?
- Try with fly.
ehmatthes commented
Uncouple Fly.io integration tests
- Move all platform-specific fly integration tests to the fly plugin dir.
- Run integration tests, see how badly it fails.
$ pytest simple_deploy/management/commands/fly_io/tests/integration_tests -x
...
import integration_tests.utils.it_helper_functions as hf
E ModuleNotFoundError: No module named 'integration_tests'
- How should a plugin handle tests, when it depends on the host for functionality?
- Maybe get the name of the plugin that's being tested?
- Maybe host has a list of plugins to always test, and a list of any installed plugins?
- Move all platform-specific integration tests to plugin dirs.
- Run integration tests with
pytest tests/integration_tests
; they should pass, because only platform-agnostic tests are left. - Try using uv, see if tests are faster while they're only running platform-agnostic tests.
- Write a
test_plugins.py
file, which specifies which plugins to run integration tests for. - Follow the traceback rabbit hole.
- Tests are outside the simple_deploy directory, so they're not installed when you install django-simple-deploy. What does that mean for testing? This only works for clones?
- Can add to
sys.path
, but maybe a better way? - Use plugin to access utils?
- Can add to
- How does a plugin access host's fixtures?
- Right now, a pretty sketchy import based on adding host conftest parent to path.
-
check_reference_file()
makes assumption that reference file is in host repo.- Use
check_reference_file_direct()
for now.
- Use
- Doesn't seem to be running
simple_deploy
during integration test for plugin. - Best road forward is probably to treat this as all in the same package, and deal with this more concretely when plugins are pulled out into external packages. There's too much hypothetical here right now.
ehmatthes commented
Notes - testing external plugins
-
test_default_plugins.py
should readpyproject.toml
to see which plugins are included by default.
ehmatthes commented
Try uv
- Run pytest with current workflow: 53 passed, 3 skipped in 18.48s, 15.81s, 16.38s.
- Install uv
to project envto system. - Replace calls to pip and venv in integration tests with
uv pip
anduv venv
. - Run pytest: 8.85s, 8.13s, 8.21s
- Update integration test to run with uv if it's available, but default to pip-based workflows otherwise.
- With all platforms' integration tests running again, this is 17.5s for a uv-enabled test run, and 45s for a pip-based run.
- On a Windows VM, that's about 55s for a uv run, and 175s for a pip-based run.
ehmatthes commented
Uncouple Fly tests - simpler approach
Just find the simplest way to make integration tests work again, with all tests in the plugin dirs, but assume they're in the same repo.
- Find a simple way to import it_helper_functions in plugin test module.
- The fixture
run_simple_deploy(autouse="module")
is not running for the plugin's integration tests.- The fixture is defined in
integration_tests/conftest.py
. It needs to be placed in a directory that's in the plugin's path as well. - Better: import it, and run it explicitly in plugin tests.
- The fixture is defined in
ehmatthes commented
Uncouple Fly e2e tests
Just make them work from within the management/commands/fly_io/
dir.
- Does
check_valid_call()
from/tests/e2e_tests/conftest.py
work?- No, omitting
-s
is allowed when it should be caught.
- No, omitting
- Move other e2e tests to plugin dirs.
- Simplify root conftest.py.
- Call platform-specific e2e tests from tests/e2e_tests.
- Sort out CLI parsing when running this way.
- Maybe call the platform test function directly, rather than trying to import it or have pytest load it?
- This may give a much more direct approach to implementing e2e tests as a plugin hook.
- Use uv if possible in e2e tests.
- Update documentation:
$ pytest tests/e2e_tests --platform fly_io -s
- Update Platform.sh test.
- Manual deployment works.
- Update Heroku test.