Uncouple tests from core

[ehmatthes]

Right now, integration and e2e tests are tightly coupled with django-simple-deploy core. Uncouple these tests, so all platform-specific integration and e2e tests are within the plugin directories.

• Pull build_dev_env.py into developer_resources. Consider developer_tools, distinct from "resources".
• Uncouple platform-specific integration tests.
• Uncouple platform-specific e2e tests.
• Run black.
• Test on Windows.
• Merge.
• Update changelog.

[ehmatthes]

Integration tests

• Make sure there's no tight coupling in:
  • Root conftest.py. Coupled through "tests/e2e_tests".
  • integration_tests/conftest.py
  • __init__.py
  • utils/it_helper_functions.py Coupled through path to reference files.
  • utils/manage_sample_projects.py. Coupled through files_dirs_to_remove.
• Can I just move a platform's tests to its plugin dir?
  • Try with fly.

[ehmatthes]

Uncouple Fly.io integration tests

• Move all platform-specific fly integration tests to the fly plugin dir.
• Run integration tests, see how badly it fails.

$ pytest simple_deploy/management/commands/fly_io/tests/integration_tests -x
...
    import integration_tests.utils.it_helper_functions as hf
E   ModuleNotFoundError: No module named 'integration_tests'

• How should a plugin handle tests, when it depends on the host for functionality?
  • Maybe get the name of the plugin that's being tested?
  • Maybe host has a list of plugins to always test, and a list of any installed plugins?
• Move all platform-specific integration tests to plugin dirs.
• Run integration tests with pytest tests/integration_tests; they should pass, because only platform-agnostic tests are left.
• Try using uv, see if tests are faster while they're only running platform-agnostic tests.
• Write a test_plugins.py file, which specifies which plugins to run integration tests for.
• Follow the traceback rabbit hole.
• Tests are outside the simple_deploy directory, so they're not installed when you install django-simple-deploy. What does that mean for testing? This only works for clones?
  • Can add to sys.path, but maybe a better way?
  • Use plugin to access utils?
• How does a plugin access host's fixtures?
  • Right now, a pretty sketchy import based on adding host conftest parent to path.
• check_reference_file() makes assumption that reference file is in host repo.
  • Use check_reference_file_direct() for now.
• Doesn't seem to be running simple_deploy during integration test for plugin.
• Best road forward is probably to treat this as all in the same package, and deal with this more concretely when plugins are pulled out into external packages. There's too much hypothetical here right now.

[ehmatthes]

Notes - testing external plugins

• test_default_plugins.py should read pyproject.toml to see which plugins are included by default.

[ehmatthes]

Try uv

• Run pytest with current workflow: 53 passed, 3 skipped in 18.48s, 15.81s, 16.38s.
• Install uv to project env to system.
• Replace calls to pip and venv in integration tests with uv pip and uv venv.
• Run pytest: 8.85s, 8.13s, 8.21s
• Update integration test to run with uv if it's available, but default to pip-based workflows otherwise.
• With all platforms' integration tests running again, this is 17.5s for a uv-enabled test run, and 45s for a pip-based run.
• On a Windows VM, that's about 55s for a uv run, and 175s for a pip-based run.

[ehmatthes]

Uncouple Fly tests - simpler approach

Just find the simplest way to make integration tests work again, with all tests in the plugin dirs, but assume they're in the same repo.

• Find a simple way to import it_helper_functions in plugin test module.
• The fixture run_simple_deploy(autouse="module") is not running for the plugin's integration tests.
  • The fixture is defined in integration_tests/conftest.py. It needs to be placed in a directory that's in the plugin's path as well.
  • Better: import it, and run it explicitly in plugin tests.

[ehmatthes]

Uncouple Fly e2e tests

Just make them work from within the management/commands/fly_io/ dir.

• Does check_valid_call() from /tests/e2e_tests/conftest.py work?
  • No, omitting -s is allowed when it should be caught.
• Move other e2e tests to plugin dirs.
• Simplify root conftest.py.
• Call platform-specific e2e tests from tests/e2e_tests.
  • Sort out CLI parsing when running this way.
  • Maybe call the platform test function directly, rather than trying to import it or have pytest load it?
  • This may give a much more direct approach to implementing e2e tests as a plugin hook.
• Use uv if possible in e2e tests.
• Update documentation: $ pytest tests/e2e_tests --platform fly_io -s
• Update Platform.sh test.
  • Manual deployment works.
• Update Heroku test.

[ehmatthes]

Closed by #332.