All imported packages seem to be tested as well

[nanne-aben]

When I run mypy on the following code, I get no errors:

import numpy

However, when I run the pytest-mypy-plugin on:

- case: test
  main: |
    import numpy
    reveal_type(1)  # N: Revealed type is "Literal[1]?"

I get a lot of errors about things that apparently go wrong within pytest and numpy:

E   pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output: 
E   Expected:
E     <45 (diff)
E   Actual:
E     ../../../../../../Users/nanneaben/opt/anaconda3/envs/tmp/lib/python3.8/site-packages/_pytest/_argcomplete:103: error: Cannot find implementation or library stub for module named "argcomplete" (diff)
E     ../../../../../../Users/nanneaben/opt/anaconda3/envs/tmp/lib/python3.8/site-packages/_pytest/_argcomplete:103: error: Cannot find implementation or library stub for module named "argcomplete.completers" (diff)
E     ../../../../../../Users/nanneaben/opt/anaconda3/envs/tmp/lib/python3.8/site-packages/_pytest/_code/code:33: error: Skipping analyzing "pluggy": found module but no type hints or library stubs (diff)
E     ../../../../../../Users/nanneaben/opt/anaconda3/envs/tmp/lib/python3.8/site-packages/_pytest/_code/code:369: error: Signature of "__getitem__" incompatible with supertype "list" (diff)
....
E     ../../../../../../Users/nanneaben/opt/anaconda3/envs/tmp/lib/python3.8/site-packages/numpy/__init__.pyi:995: error: Overloaded function signatures 22 and 51 overlap with incompatible return types (diff)
E     ../../../../../../Users/nanneaben/opt/anaconda3/envs/tmp/lib/python3.8/site-packages/numpy/__init__.pyi:997: error: Overloaded function signatures 23 and 51 overlap with incompatible return types (diff)
E     ../../../../../../Users/nanneaben/opt/anaconda3/envs/tmp/lib/python3.8/site-packages/numpy/__init__.pyi:999: error: Overloaded function signatures 24 and 51 overlap with incompatible return types (diff)
E     ../../../../../../Users/nanneaben/opt/anaconda3/envs/tmp/lib/python3.8/site-packages/numpy/__init__.pyi:1001: error: Overloaded function signatures 25 and 51 overlap with incompatible return types (diff)
E     ../../../../../../Users/nanneaben/opt/anaconda3/envs/tmp/lib/python3.8/site-packages/numpy/__init__.pyi:1003: error: Overloaded function signatures 26 and 51 overlap with incompatible return types (diff)
...

Is this the expected behaviour? Am I doing something wrong? Is it possible to make pytest-mypy-plugin match the behaviour of mypy here?

The above is a simplified example, of course. The real use-case here is that I'd like to use pytest-mypy-plugin to typecheck my package, but my package imports numpy (as well as some other packages that seem to show this behaviour), so currently all tests that import my package result in an error.

Thanks in advance!

[sobolevn]

See #34

[nanne-aben]

Thanks, using pytest --mypy-only-local-stub indeed resolves the issue!