F0002 When trying to infer too many items
tushar-deepsource opened this issue · 4 comments
Bug description
Pylint throws an F0002 while trying to lint a file with large enough inferences, here's a minimal example:
x0, x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11, x12, x13, x14, x15, x16, x17, x18, x19, x20, x21, x22, x23, x24, x25, x26, x27, x28, x29, x30 = foo()
def bar():
return [
x0 + x1 + x10 + x100 + x101 + x103 + x104 + x106 + x109 + x11,
x0 + x1 + x10 + x100 + x101 + x102 + x103 + x104 + x105 + x106 + x107 + x108 + x109 + x11 + x110 + x111 + x112 + x113 + x114 + x115 + x116 + x117 + x118 + x119 + x12 + x120 + x121 + x122 + x123 + x124 + x125 + x126 + x127 + x128 + x129 + x13 + x130 + x131 + x132 + x133 + x134 + x135 + x136 + x137 + x138 + x139 + x14 + x140 + x141 + x142 + x143 + x144 + x145 + x146 + x147 + x148 + x149 + x15 + x150 + x151 + x152 + x153 + x154 + x155 + x156 + x157 + x158 + x159 + x16 + x160 + x161 + x162 + x163 + x164 + x17 + x18 + x19 + x2 + x20 + x21 + x22 + x23 + x24 + x25 + x26 + x27 + x28 + x29 + x3 + x30 + x31 + x32 + x33 + x34 + x35 + x36 + x37 + x38 + x39 + x4 + x40 + x41 + x42 + x43 + x44 + x45 + x46 + x47 + x48 + x49 + x5 + x50 + x51 + x52 + x53 + x54 + x55 + x56 + x57 + x58 + x59 + x6 + x60 + x61 + x62 + x63 + x64 + x65 + x66 + x67 + x68 + x69 + x7 + x70 + x71 + x72 + x73 + x8 + x9
]
es = bar()
print([e.foobar() for e in es])
This can be handled and caught in astroid during inference, rather than reporting it to the user.
Configuration
No response
Command used
pylint asd.py
Pylint output
Exception on node <Call l.11 at 0x102a5b4c0> in file '/Users/tusharsadhwani/code/marvin-python/asd.py'
Traceback (most recent call last):
File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/decorators.py", line 140, in raise_if_nothing_inferred
yield next(generator)
File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/decorators.py", line 104, in wrapped
if context.push(node):
RecursionError: maximum recursion depth exceeded
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/pylint/checkers/utils.py", line 1395, in infer_all
return list(node.infer(context=context))
File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/nodes/node_ng.py", line 169, in infer
yield from self._infer(context=context, **kwargs)
File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/decorators.py", line 154, in raise_if_nothing_inferred
yield from generator
File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/decorators.py", line 109, in wrapped
for res in _func(node, context, **kwargs):
File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/inference.py", line 344, in infer_attribute
for owner in self.expr.infer(context):
File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/nodes/node_ng.py", line 182, in infer
for i, result in enumerate(self._infer(context=context, **kwargs)):
File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/decorators.py", line 154, in raise_if_nothing_inferred
yield from generator
File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/decorators.py", line 109, in wrapped
for res in _func(node, context, **kwargs):
File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/bases.py", line 165, in _infer_stmts
for inf in stmt.infer(context=context): # type: ignore[union-attr]
File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/nodes/node_ng.py", line 182, in infer
for i, result in enumerate(self._infer(context=context, **kwargs)):
[..... TRUNCATED .....]
File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/inference.py", line 874, in _infer_binop
for lhs, rhs in itertools.product(lhs_iter, rhs_iter):
File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/nodes/node_ng.py", line 182, in infer
for i, result in enumerate(self._infer(context=context, **kwargs)):
File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/decorators.py", line 127, in yes_if_nothing_inferred
yield next(generator)
File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/decorators.py", line 109, in wrapped
for res in _func(node, context, **kwargs):
File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/inference.py", line 532, in _filter_operation_errors
for result in infer_callable(self, context):
File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/inference.py", line 874, in _infer_binop
for lhs, rhs in itertools.product(lhs_iter, rhs_iter):
File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/nodes/node_ng.py", line 182, in infer
for i, result in enumerate(self._infer(context=context, **kwargs)):
File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/decorators.py", line 150, in raise_if_nothing_inferred
raise InferenceError(
File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/exceptions.py", line 277, in __init__
super().__init__(message, **kws)
RecursionError: maximum recursion depth exceeded while calling a Python object
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/pylint/utils/ast_walker.py", line 90, in walk
callback(astroid)
File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/pylint/checkers/base/basic_error_checker.py", line 432, in visit_call
for inferred in infer_all(node.func):
File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/pylint/checkers/utils.py", line 1399, in infer_all
raise AstroidError from e
astroid.exceptions.AstroidError
asd.py:1:0: F0002: asd.py: Fatal error while checking 'asd.py'. Please open an issue in our bug tracker so we address this. There is a pre-filled template that you can use in '/Users/tusharsadhwani/Library/Caches/pylint/pylint-crash-2022-12-05-16-15-46.txt'. (astroid-error)
Expected behavior
No F0002
Pylint version
pylint 2.15.7
astroid 2.12.13
Python 3.10.6 (main, Aug 30 2022, 04:58:14) [Clang 13.1.6 (clang-1316.0.21.2.5)]
OS / Environment
No response
Additional dependencies
No response
We had this problem several time, but I'm not sure that silently failing all the time with an inference error is better for genuinely large recursions or genuinely big calculation. Maybe we should catch the RecursionError or use a timeout, but raise a warning...
Yeah, sounds good to me.
I have often voted against catching RecursionErrors
too broadly. Often it is just a sign of poorly written code, although I must confess in this case I don't see an immediate fix.
What doesn't help is that I think the calls to all the decorators that we use in astroid
make us hit the error sooner?
This specific example no longer reproduces on pylint 3.0.3. We have other open issues about recursion where we can take up the question of what to do, e.g. #8842.
I have often voted against catching RecursionErrors too broadly. Often it is just a sign of poorly written code, although I must confess in this case I don't see an immediate fix.
We want pylint to be used on code that needs help. It won't ever be adopted if it crashes when people try to first use it. The fact that astroid
uses recursive algorithms is an implementation detail we shouldn't blame on the code being linted.