Doubles's pytest plugin reports failed assertion as errors rather than failed tests

[charlax]

Hey Todd,

Hope you're doing great!

With Python 3.6 and latest pytest, failed expectations show up in the error portion of pytests' report, rather than under the specific tests failure details. This makes it a bit difficult to understand which test actually failed, because by default pytest only shows the test name.

Concretely, it shows up like this:

============================================== ERRORS ===============================================
_________________________________ ERROR at teardown of test_update __________________________________
Traceback (most recent call last):
  File "/Users/ca/.local/share/virtualenvs/api-umOVruWb/src/doubles/doubles/pytest_plugin.py", line 9, in verify_and_teardown_doubles
    verify()
  File "/Users/ca/.local/share/virtualenvs/api-umOVruWb/src/doubles/doubles/lifecycle.py", line 52, in verify
    _thread_local_data.current_space.verify()
  File "/Users/ca/.local/share/virtualenvs/api-umOVruWb/src/doubles/doubles/space.py", line 73, in verify
    proxy.verify()
  File "/Users/ca/.local/share/virtualenvs/api-umOVruWb/src/doubles/doubles/proxy.py", line 56, in verify
    method_double.verify()
  File "/Users/ca/.local/share/virtualenvs/api-umOVruWb/src/doubles/doubles/method_double.py", line 66, in verify
    expectation.raise_failure_exception()
  File "/Users/ca/.local/share/virtualenvs/api-umOVruWb/src/doubles/doubles/expectation.py", line 77, in raise_failure_exception
    super(Expectation, self).raise_failure_exception('Expected')
  File "/Users/ca/.local/share/virtualenvs/api-umOVruWb/src/doubles/doubles/allowance.py", line 342, in raise_failure_exception
    self._caller.lineno,
doubles.exceptions.MockExpectationError: Expected 'update' to be called on <InstanceDouble of <class 'whatever'> object at 0x112c15c88> with (User(id=UUID('24259817-d1bf-4a65-9435-7e8c71372340'), firstname='Sean', lastname='', email='jcombs@thompson.com'), <DocField name="title" document_id="None" value="Richard Murray">), but was not. (test_whatever.py:19)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ca/.local/share/virtualenvs/api-umOVruWb/src/doubles/doubles/pytest_plugin.py", line 11, in verify_and_teardown_doubles
    raise pytest.Failed(str(e)) from None
AttributeError: module 'pytest' has no attribute 'Failed'
================================ 38 passed, 1 error in 2.53 seconds =================================

(fake data from factory.Faker)

A potential solution would be to take inspiration from https://github.com/pytest-dev/pytest-mock/blob/master/pytest_mock.py

[toddsifleet]

Thanks for reporting @charlax, I hope things are going well on your end.

I will take a look at getting this fixed over the weekend.

[charlax]

Thanks Todd! Things are great on my end :)

I'm reopening though, this is still happening with:

This is pytest version 3.5.0, imported from /Users/ca/.local/share/virtualenvs/api-umOVruWb/lib/python3.6/site-packages/pytest.py
setuptools registered plugins:
  pytest-flask-0.10.0 at /Users/ca/.local/share/virtualenvs/api-umOVruWb/lib/python3.6/site-packages/pytest_flask/plugin.py
  pytest-factoryboy-2.0.1 at /Users/ca/.local/share/virtualenvs/api-umOVruWb/lib/python3.6/site-packages/pytest_factoryboy/plugin.py
  doubles-1.3.2 at /Users/ca/.local/share/virtualenvs/api-umOVruWb/lib/python3.6/site-packages/doubles/pytest_plugin.py

[charlax]

(looks like I can't reopen the issue)

Here's how it's displayed for me:

============================================== ERRORS ===============================================
_________________________________ ERROR at teardown of test_accept __________________________________
Traceback (most recent call last):
  File "/Users/ca/.local/share/virtualenvs/api-umOVruWb/lib/python3.6/site-packages/doubles/pytest_plugin.py", line 8, in verify_and_teardown_doubles
    verify()
  File "/Users/ca/.local/share/virtualenvs/api-umOVruWb/lib/python3.6/site-packages/doubles/lifecycle.py", line 52, in verify
    _thread_local_data.current_space.verify()
  File "/Users/ca/.local/share/virtualenvs/api-umOVruWb/lib/python3.6/site-packages/doubles/space.py", line 73, in verify
    proxy.verify()
  File "/Users/ca/.local/share/virtualenvs/api-umOVruWb/lib/python3.6/site-packages/doubles/proxy.py", line 56, in verify
    method_double.verify()
  File "/Users/ca/.local/share/virtualenvs/api-umOVruWb/lib/python3.6/site-packages/doubles/method_double.py", line 66, in verify
    expectation.raise_failure_exception()
  File "/Users/ca/.local/share/virtualenvs/api-umOVruWb/lib/python3.6/site-packages/doubles/expectation.py", line 77, in raise_failure_exception
    super(Expectation, self).raise_failure_exception('Expected')
  File "/Users/ca/.local/share/virtualenvs/api-umOVruWb/lib/python3.6/site-packages/doubles/allowance.py", line 342, in raise_failure_exception
    self._caller.lineno,
doubles.exceptions.MockExpectationError: Expected 'accept' to be called on [redacted]
================================ x passed, 1 error in x seconds =================================

[charlax]

(btw for some reason I don't receive email notification for this thread, so feel free to drop me an email if you want me to test something)

[toddsifleet]

Unfortunately @charlax I think this is expected, see test code:

Looking at pytest documentation I didn't see an easy way to actually fail the test in the pytest finalizer. I will try to connect with the pytest folks to see if they have an idea.