Unclear error mocking a class and asserting called with

[justinessert]

Summary

I am noticed some issues with mocking, or at the very least there is an unclear error message. The issue occurs when the following conditions apply (although there may be additional conditions that I missed):

1. There is a class that you want to mock.
2. You create a MagicMock instance of that class.
3. You call that MagicMock instance with a set of invalid parameters for the real class (eg. if the class requires an a parameter, but you don't pass it).
4. You try to assert that the MagicMock instance was called with those invalid parameters.

Example

Example Code

This is easier to explain via code though, so for this test I am trying to mock the following class:

class AClass:
    def __init__(self, a, b=2):
        pass

And I am running the following test (using different values for kwargs):

def test_mock(kwargs):
    mock_a_class = MagicMock(AClass)
    mock_a_class(**kwargs)
    mock_a_class.assert_called_once_with(**kwargs)

Expected Behavior

Initially I would expect this test to pass for any kwargs dictionary, since I am just calling a mocked object with a set of kwargs and then testing that it was called with that same set of kwargs.

Alternatively, I would also understand if the mock_a_class(**kwargs) in the same way that AClass(**kwargs) would fail if the kwargs are not valid.

Actual Behavior

If the kwargs dictionary is an invalid set of parameters for the AClass initialization method, then the mock_a_class.assert_called_once_with(**kwargs) line fails (although the mock_a_class(**kwargs) line succeeds).

So for the above example, the following values of kwargs gives a successful result:

• dict(a=1)
• dict(a=1, b=2)

And these are values of kwargs that lead to a failed result:

• dict()
• dict(c=3)
• dict(a=1, c=3)
• dict(a=1, b=2, c=3)

Additionally, the error message that you get for these failed cases doesn't tell you anything useful. For example, the error message when kwargs=dict() is:

E           AssertionError: expected call not found.
E           Expected: mock()
E           Actual: mock()

This made debugging the issue very hard and frustrating. At the very least, can we at least have a clearer error message when this happens?

pytest-mock version: 3.10.0

[cjw296]

This package is a rolling backport of unittest.mock.
As such, any problems you encounter most likely need to be fixed upstream.

Please can you try and reproduce the problem on the latest release of Python 3, including alphas, and replace any import from mock with ones from unittest.mock.

If the issue still occurs, then please report upstream through https://github.com/python/cpython/issues as it will need to be fixed there so that it can be backported here and released to you.

If not, reply with what you find out and we can re-open this issue.

[tirkarthi]

I can confirm that the issue exists in master branch of CPython. The issue is that using spec only checks for invalid attribute access and not signatures. So MagicMock(AClass) uses AClass as spec and checks for only attribute access on the mock allowing invalid attributes but assert_ helpers check for signature since Python 3.4. There was a proposal to make spec also do signature checking but it will break backwards compatibility due to strict checks and auto spec is also expensive since it would also apply it for methods. I would suggest using create_autospec since that will check for signature too catching these issues earlier.

from unittest.mock import MagicMock, create_autospec

class AClass:

    def __init__(self, a, b=2):
        pass


def test_mock(kwargs):
    mock_a_class = create_autospec(AClass)
    mock_a_class(**kwargs)
    mock_a_class.assert_called_once_with(**kwargs)

test_mock({})
test_mock({'c': 1, 'a': 2})

python /tmp/foo.py
Traceback (most recent call last):
  File "/tmp/foo.py", line 14, in <module>
    test_mock({})
  File "/tmp/foo.py", line 11, in test_mock
    mock_a_class(**kwargs)
  File "/usr/lib/python3.10/unittest/mock.py", line 1102, in __call__
    self._mock_check_sig(*args, **kwargs)
  File "/usr/lib/python3.10/unittest/mock.py", line 123, in checksig
    sig.bind(*args, **kwargs)
  File "/usr/lib/python3.10/inspect.py", line 3179, in bind
    return self._bind(args, kwargs)
  File "/usr/lib/python3.10/inspect.py", line 3094, in _bind
    raise TypeError(msg) from None
TypeError: missing a required argument: 'a'

python/cpython#74772