test failures are really really difficult to read
asottile opened this issue · 1 comments
asottile commented
here's an example failure I had to debug recently:
$ pip install . -qq --no-deps && python -m pytest tests -k test_emits_error_for_unresolved_managers
============================= test session starts ==============================
platform linux -- Python 3.10.12, pytest-8.3.1, pluggy-1.5.0 -- /tmp/django-stubs/venv/bin/python
cachedir: .pytest_cache
rootdir: /tmp/django-stubs
configfile: pytest.ini
plugins: mypy-plugins-3.1.2, shard-0.1.2
collected 415 items / 414 deselected / 1 selected
Running 1 items in this shard: tests/typecheck/managers/test_managers.yml::test_emits_error_for_unresolved_managers
tests/typecheck/managers/test_managers.yml::test_emits_error_for_unresolved_managers FAILED
=================================== FAILURES ===================================
___________________ test_emits_error_for_unresolved_managers ___________________
/tmp/django-stubs/tests/typecheck/managers/test_managers.yml:495:
E pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output:
E Actual:
E ...
E myapp/models:36: note: Revealed type is "django.db.models.manager.Manager[myapp.models.User]" (diff)
E myapp/models:37: note: Revealed type is "django.db.models.manager.Manager[myapp.models.User]" (diff)
E myapp/models:39: note: Revealed type is "myapp.models.UnknownManager[myapp.models.Booking]" (diff)
E myapp/models:40: note: Revealed type is "django.db.models.manager.Manager[myapp.models.Booking]" (diff)
E myapp/models:42: note: Revealed type is "myapp.models.UnknownManager[myapp.models.TwoUnresolvable]" (diff)
E myapp/models:43: note: Revealed type is "myapp.models.UnknownManager[myapp.models.TwoUnresolvable]" (diff)
E myapp/models:44: note: Revealed type is "django.db.models.manager.Manager[myapp.models.TwoUnresolvable]" (diff)
E myapp/models:46: note: Revealed type is "myapp.models.UnknownManager[myapp.models.AbstractUnresolvable]" (diff)
E myapp/models:47: note: Revealed type is "django.db.models.manager.Manager[myapp.models.InvisibleUnresolvable]" (diff)
E myapp/models:49: note: Revealed type is "django.db.models.fields.related_descriptors.RelatedManager[myapp.models.Booking]" (diff)
E myapp/models:50: note: Revealed type is "django.db.models.fields.related_descriptors.RelatedManager[myapp.models.Booking]" (diff)
E myapp/models:53: note: Revealed type is "def () -> myapp.models.UnknownQuerySet[myapp.models.Booking, myapp.models.Booking]" (diff)
E myapp/models:54: note: Revealed type is "Any" (diff)
E myapp/models:55: note: Revealed type is "def (*args: Any, **kwargs: Any) -> myapp.models.UnknownQuerySet[myapp.models.Booking, myapp.models.Booking]" (diff)
E myapp/models:56: note: Revealed type is "Any" (diff)
E myapp/models:57: note: Revealed type is "Union[myapp.models.Booking, None]" (diff)
E myapp/models:58: note: Revealed type is "myapp.models.Booking" (diff)
E myapp/models:59: note: Revealed type is "builtins.list[myapp.models.Booking]" (diff)
E myapp/models:60: note: Revealed type is "builtins.list[myapp.models.Booking]" (diff)
E myapp/models:64: note: Revealed type is "def () -> django.db.models.query.QuerySet[myapp.models.Booking, myapp.models.Booking]" (diff)
E myapp/models:65: error: "RelatedManager[Booking]" has no attribute "custom" [attr-defined] (diff)
E myapp/models:65: note: Revealed type is "Any" (diff)
E myapp/models:66: note: Revealed type is "def (*args: Any, **kwargs: Any) -> django.db.models.query.QuerySet[myapp.models.Booking, myapp.models.Booking]" (diff)
E myapp/models:67: error: "QuerySet[Booking, Booking]" has no attribute "custom" [attr-defined] (diff)
E myapp/models:67: note: Revealed type is "Any" (diff)
E myapp/models:68: note: Revealed type is "Union[myapp.models.Booking, None]" (diff)
E Expected:
E ...
E myapp/models:32: error: Could not resolve manager type for "myapp.models.InvisibleUnresolvable.objects" [django-manager-missing] (diff)
E myapp/models:36: note: Revealed type is "django.db.models.manager.Manager[myapp.models.User]" (diff)
E myapp/models:37: note: Revealed type is "django.db.models.manager.Manager[myapp.models.User]" (diff)
E myapp/models:39: note: Revealed type is "myapp.models.UnknownManager[myapp.models.Booking]" (diff)
E myapp/models:40: note: Revealed type is "django.db.models.manager.Manager[myapp.models.Booking]" (diff)
E myapp/models:42: note: Revealed type is "myapp.models.UnknownManager[myapp.models.TwoUnresolvable]" (diff)
E myapp/models:43: note: Revealed type is "myapp.models.UnknownManager[myapp.models.TwoUnresolvable]" (diff)
E myapp/models:44: note: Revealed type is "django.db.models.manager.Manager[myapp.models.TwoUnresolvable]" (diff)
E myapp/models:46: note: Revealed type is "myapp.models.UnknownManager[myapp.models.InvisibleUnresolvable]" (diff)
E myapp/models:47: note: Revealed type is "django.db.models.manager.Manager[myapp.models.InvisibleUnresolvable]" (diff)
E myapp/models:49: note: Revealed type is "django.db.models.fields.related_descriptors.RelatedManager[myapp.models.Booking]" (diff)
E myapp/models:50: note: Revealed type is "django.db.models.fields.related_descriptors.RelatedManager[myapp.models.Booking]" (diff)
E myapp/models:53: note: Revealed type is "def () -> myapp.models.UnknownQuerySet[myapp.models.Booking, myapp.models.Booking]" (diff)
E myapp/models:54: note: Revealed type is "Any" (diff)
E myapp/models:55: note: Revealed type is "def (*args: Any, **kwargs: Any) -> myapp.models.UnknownQuerySet[myapp.models.Booking, myapp.models.Booking]" (diff)
E myapp/models:56: note: Revealed type is "Any" (diff)
E myapp/models:57: note: Revealed type is "Union[myapp.models.Booking, None]" (diff)
E myapp/models:58: note: Revealed type is "myapp.models.Booking" (diff)
E myapp/models:59: note: Revealed type is "builtins.list[myapp.models.Booking]" (diff)
E myapp/models:60: note: Revealed type is "builtins.list[myapp.models.Booking]" (diff)
E myapp/models:64: note: Revealed type is "def () -> django.db.models.query.QuerySet[myapp.models.Booking, myapp.models.Booking]" (diff)
E myapp/models:65: error: "RelatedManager[Booking]" has no attribute "custom" [attr-defined] (diff)
E myapp/models:65: note: Revealed type is "Any" (diff)
E myapp/models:66: note: Revealed type is "def (*args: Any, **kwargs: Any) -> django.db.models.query.QuerySet[myapp.models.Booking, myapp.models.Booking]" (diff)
E myapp/models:67: error: "QuerySet[Booking, Booking]" has no attribute "custom" [attr-defined] (diff)
E myapp/models:67: note: Revealed type is "Any" (diff)
E myapp/models:68: note: Revealed type is "Union[myapp.models.Booking, None]" (diff)
E Alignment of first line difference:
E E: myapp/models:32: error: Could not resolve manager type for "myapp.models...
E A: myapp/models:36: note: Revealed type is "django.db.models.manager.Manage...
E ^
=========================== short test summary info ============================
FAILED tests/typecheck/managers/test_managers.yml::test_emits_error_for_unresolved_managers -
====================== 1 failed, 414 deselected in 3.89s =======================
the "Alignment of first line difference" part is "useful" at least, but every line is marked as (diff)
-- I ended up copying Actual
and Expected
to a file and using diff -u
:
$ diff -u expected actual
--- expected 2024-07-25 15:31:06.184912437 -0400
+++ actual 2024-07-25 15:31:08.888821824 -0400
@@ -1,4 +1,3 @@
-E myapp/models:32: error: Could not resolve manager type for "myapp.models.InvisibleUnresolvable.objects" [django-manager-missing] (diff)
E myapp/models:36: note: Revealed type is "django.db.models.manager.Manager[myapp.models.User]" (diff)
E myapp/models:37: note: Revealed type is "django.db.models.manager.Manager[myapp.models.User]" (diff)
E myapp/models:39: note: Revealed type is "myapp.models.UnknownManager[myapp.models.Booking]" (diff)
@@ -6,7 +5,7 @@
E myapp/models:42: note: Revealed type is "myapp.models.UnknownManager[myapp.models.TwoUnresolvable]" (diff)
E myapp/models:43: note: Revealed type is "myapp.models.UnknownManager[myapp.models.TwoUnresolvable]" (diff)
E myapp/models:44: note: Revealed type is "django.db.models.manager.Manager[myapp.models.TwoUnresolvable]" (diff)
-E myapp/models:46: note: Revealed type is "myapp.models.UnknownManager[myapp.models.InvisibleUnresolvable]" (diff)
+E myapp/models:46: note: Revealed type is "myapp.models.UnknownManager[myapp.models.AbstractUnresolvable]" (diff)
E myapp/models:47: note: Revealed type is "django.db.models.manager.Manager[myapp.models.InvisibleUnresolvable]" (diff)
E myapp/models:49: note: Revealed type is "django.db.models.fields.related_descriptors.RelatedManager[myapp.models.Booking]" (diff)
E myapp/models:50: note: Revealed type is "django.db.models.fields.related_descriptors.RelatedManager[myapp.models.Booking]" (diff)
much much more readable! and at least as a programmer I'm used to looking at unified diffs and would have been able to spot the difference much quicker!
sobolevn commented
Yes, this is really hard. It took me several minutes just to review one line change (I had to count lines separately, because lines in IDE do not match lines in main.py
example).
I am open to suggestions.