IndexError
Dreamsorcerer opened this issue ยท 7 comments
With this file:
for i in seq_a:
for j in seq_b:
print(i, j)
I get this output:
Traceback (most recent call last):
File "/home/ubuntu/.local/bin/flake8", line 11, in <module>
sys.exit(main())
File "/home/ubuntu/.local/lib/python3.6/site-packages/flake8/main/cli.py", line 18, in main
app.run(argv)
File "/home/ubuntu/.local/lib/python3.6/site-packages/flake8/main/application.py", line 393, in run
self._run(argv)
File "/home/ubuntu/.local/lib/python3.6/site-packages/flake8/main/application.py", line 381, in _run
self.run_checks()
File "/home/ubuntu/.local/lib/python3.6/site-packages/flake8/main/application.py", line 300, in run_checks
self.file_checker_manager.run()
File "/home/ubuntu/.local/lib/python3.6/site-packages/flake8/checker.py", line 331, in run
self.run_serial()
File "/home/ubuntu/.local/lib/python3.6/site-packages/flake8/checker.py", line 315, in run_serial
checker.run_checks()
File "/home/ubuntu/.local/lib/python3.6/site-packages/flake8/checker.py", line 598, in run_checks
self.run_ast_checks()
File "/home/ubuntu/.local/lib/python3.6/site-packages/flake8/checker.py", line 502, in run_ast_checks
for (line_number, offset, text, check) in runner:
File "/home/ubuntu/.local/lib/python3.6/site-packages/flake8_aaa/checker.py", line 48, in run
self.load()
File "/home/ubuntu/.local/lib/python3.6/site-packages/flake8_aaa/checker.py", line 37, in load
self.ast_tokens = asttokens.ASTTokens(''.join(self.lines), tree=self.tree)
File "/home/ubuntu/.local/lib/python3.6/site-packages/asttokens/asttokens.py", line 65, in __init__
self.mark_tokens(self._tree)
File "/home/ubuntu/.local/lib/python3.6/site-packages/asttokens/asttokens.py", line 76, in mark_tokens
MarkTokens(self).visit_tree(root_node)
File "/home/ubuntu/.local/lib/python3.6/site-packages/asttokens/mark_tokens.py", line 49, in visit_tree
util.visit_tree(node, self._visit_before_children, self._visit_after_children)
File "/home/ubuntu/.local/lib/python3.6/site-packages/asttokens/util.py", line 199, in visit_tree
ret = postvisit(current, par_value, value)
File "/home/ubuntu/.local/lib/python3.6/site-packages/asttokens/mark_tokens.py", line 92, in _visit_after_children
nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
File "/home/ubuntu/.local/lib/python3.6/site-packages/asttokens/mark_tokens.py", line 189, in handle_attr
name = self._code.next_token(dot)
File "/home/ubuntu/.local/lib/python3.6/site-packages/asttokens/asttokens.py", line 141, in next_token
while is_non_coding_token(self._tokens[i].type):
IndexError: list index out of range
Hi @Dreamsorcerer - thanks for opening this issue.
Looking at the last line of the stack trace, it seems like this is an error in asttokens (if you're interested, that's this library that we use to help analyse the Python AST https://github.com/gristlabs/asttokens)...
File "/home/ubuntu/.local/lib/python3.6/site-packages/asttokens/asttokens.py", line 141, in next_token
while is_non_coding_token(self._tokens[i].type):
Could you help me by confirming:
- You're on Python 3.6 - is that right? Could you give the output of
python --version
? - Could you share the versions of Flake8, Flake8-AAA and asttokens that are installed? Doing
pip freeze
and sharing the output would be helpful.
Thanks ๐
I'm not currently able to reproduce with Python 3.7 and Flake8-AAA v0.9.0:
$ pip install flake8-aaa flake8
Collecting flake8-aaa
Using cached flake8_aaa-0.9.0-py3-none-any.whl (16 kB)
Collecting flake8
Using cached flake8-3.7.9-py2.py3-none-any.whl (69 kB)
Collecting asttokens>=2
Downloading asttokens-2.0.4-py2.py3-none-any.whl (20 kB)
Collecting entrypoints<0.4.0,>=0.3.0
Using cached entrypoints-0.3-py2.py3-none-any.whl (11 kB)
Collecting mccabe<0.7.0,>=0.6.0
Using cached mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB)
Collecting pyflakes<2.2.0,>=2.1.0
Using cached pyflakes-2.1.1-py2.py3-none-any.whl (59 kB)
Collecting pycodestyle<2.6.0,>=2.5.0
Using cached pycodestyle-2.5.0-py2.py3-none-any.whl (51 kB)
Collecting six
Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Installing collected packages: six, asttokens, flake8-aaa, entrypoints, mccabe, pyflakes, pycodestyle, flake8
Successfully installed asttokens-2.0.4 entrypoints-0.3 flake8-3.7.9 flake8-aaa-0.9.0 mccabe-0.6.1 pycodestyle-2.5.0 pyflakes-2.1.1 six-1.14.0
$ cat > test.py
for i in seq_a:
for j in seq_b:
print(i, j)
$ flake8 test.py
test.py:1:10: F821 undefined name 'seq_a'
test.py:2:14: F821 undefined name 'seq_b'
Instead, Flake8 just errors that seq_a
and seq_b
are not defined. Hopefully the answers to the questions above will help me reproduce. ๐ค
This is after uninstalling flake8-aaa, in case you wonder why it's not in there.
> python3 --version
Python 3.6.8
> flake8 --version
3.7.9 (assertive: 1.2.1, flake8-annotations: 2.1.0, flake8-annotations-complexity: 0.0.4, flake8-bandit: 2.1.2, flake8-broken-line: 0.2.0, flake8-bugbear: 20.1.4, flake8-class-attributes-order: 0.1.0, flake8-comprehensions: 3.2.2, flake8-darglint: 0.4.1, flake8-debugger: 3.2.1, flake8-docstrings: 1.5.0, pydocstyle: 5.0.2, flake8-eradicate: 0.3.0, flake8-executable: 2.0.3, flake8-requirements: 1.3.1, flake8-sorted-keys: 0.2.0, flake8-string-format: 0.3.0, flake8-todos: 1.0.0, flake8.datetimez: 19.5.4.0, flake8_commas: 2.0.0, flake8_isort: 2.9.1, flake8_quotes: 3.0.0, good-smell: 0.1, import-order: 0.18.1, logging-format: 0.6.0, mccabe: 0.6.1, naming: 0.10.0, pycodestyle: 2.5.0, pyflakes: 2.1.1, radon: 4.1.0, rst-docstrings: 0.0.12, wemake_python_styleguide: 0.14.0) CPython 3.6.8 on Linux
> pip3 freeze
acme==0.26.1
Aggregated-Notifications-API==1.0
aiohttp==3.6.2
aiohttp-debugtoolbar==0.6.0
aiohttp-devtools==0.13.1
aiohttp-jinja2==1.2.0
aiohttp-security==0.4.0
aiohttp-session==2.9.0
asn1crypto==0.24.0
astor==0.8.1
astpretty==2.0.0
asttokens==2.0.4
async-timeout==3.0.1
asyncio-glib==0.1
attrs==19.3.0
bandit==1.6.2
beautifulsoup4==4.6.3
certbot==0.26.0.dev0
certbot-nginx==0.23.0
certifi==2019.11.28
cffi==1.14.0
chardet==3.0.4
click==7.1.2
clickclick==1.2.2
cognitive-complexity==0.0.4
colorama==0.4.3
ConfigArgParse==0.11.0
configobj==5.0.6
connexion==1.5.3
contractions==0.0.17
cryptography==2.9.2
Cython==0.29.14
darglint==1.2.3
dataclasses==0.6
devtools==0.5.1
distro-info===0.18ubuntu0.18.04.1
docutils==0.16
entrypoints==0.3
eradicate==1.0
fire==0.1.3
flake8==3.7.9
flake8-annotations==2.1.0
flake8-annotations-complexity==0.0.4
flake8-assertive==1.2.1
flake8-bandit==2.1.2
flake8-broken-line==0.2.0
flake8-bugbear==20.1.4
flake8-class-attributes-order==0.1.0
flake8-commas==2.0.0
flake8-comprehensions==3.2.2
flake8-datetimez==19.5.4.0
flake8-debugger==3.2.1
flake8-docstrings==1.5.0
flake8-eradicate==0.3.0
flake8-executable==2.0.3
flake8-import-order==0.18.1
flake8-isort==2.9.1
flake8-logging-format==0.6.0
flake8-plugin-utils==1.0.0
flake8-polyfill==1.0.2
flake8-quotes==3.0.0
flake8-requirements==1.3.1
flake8-rst-docstrings==0.0.12
flake8-sorted-keys==0.2.0
flake8-string-format==0.3.0
flake8-todos==0.1.3
Flask==1.0.2
future==0.18.2
gbulb==0.6.1
gitdb==4.0.4
gitdb2==2.0.6
GitPython==3.1.1
good-smell==0.13.0
idna==2.9
idna-ssl==1.1.0
importlib-metadata==1.6.0
inflect==2.1.0
inflection==0.3.1
isort==4.3.21
itsdangerous==0.24
Jinja2==2.11.2
josepy==1.1.0
jsonschema==2.6.0
keyring==10.6.0
keyrings.alt==3.0
lz4==3.0.2
Mako==1.0.7
mando==0.6.4
MarkupSafe==1.1.1
mccabe==0.6.1
mock==2.0.0
more-itertools==8.1.0
multidict==4.7.5
mypy==0.770
mypy-extensions==0.4.3
netifaces==0.10.4
nltk==3.4
outcome==1.0.1
parsedatetime==2.4
pbr==5.4.5
pep8-naming==0.10.0
pluggy==0.6.0
py==1.5.2
pycairo==1.18.2
pycodestyle==2.5.0
pycparser==2.20
pycrypto==2.6.1
pydocstyle==5.0.2
pyflakes==2.1.1
Pygments==2.6.1
PyGObject==3.34.0
pynput==1.4.2
pyOpenSSL==17.5.0
pyparsing==2.2.0
pyRFC3339==1.0
python-apt==1.6.5+ubuntu0.2
python-debian==0.1.32
python-xlib==0.25
pytz==2018.3
pyxdg==0.25
PyYAML==5.3.1
radon==4.1.0
Represent==1.6.0
requests==2.22.0
requests-toolbelt==0.8.0
restructuredtext-lint==1.3.0
SecretStorage==2.3.1
singledispatch==3.4.0.3
six==1.14.0
smmap==3.0.2
smmap2==2.0.5
snowballstemmer==2.0.0
SQLAlchemy==1.3.16
sqlalchemy-aio==0.15.0
sqlparse==0.3.1
stevedore==1.32.0
swagger-spec-validator==2.4.1
testfixtures==6.14.1
toml==0.10.0
tornado==5.1.1
tox==2.5.0
trafaret==2.0.2
typed-ast==1.4.1
typing-extensions==3.7.4.2
unattended-upgrades==0.1
urllib3==1.25.7
virtualenv==15.1.0
watchgod==0.6
wemake-python-styleguide==0.14.0
Werkzeug==0.14.1
yarl==1.4.2
zipp==3.1.0
zope.component==4.3.0
zope.event==4.2.0
zope.hookable==4.0.4
zope.interface==4.3.2
Thanks for providing that pip freeze
output - you've got a lot of packages installed!
Firstly, the good news is that Python 3.6 works with the example snippet:
$ python --version
Python 3.6.7
$ pip install flake8 flake8-aaa
...
Successfully installed asttokens-2.0.4 entrypoints-0.3 flake8-3.7.9 flake8-aaa-0.9.0 mccabe-0.6.1 pycodestyle-2.5.0 pyflakes-2.1.1 six-1.14.0
$ cat > test.py
for i in seq_a:
for j in seq_b:
print(i, j)
$ flake8 test.py
test.py:1:10: F821 undefined name 'seq_a'
test.py:2:14: F821 undefined name 'seq_b'
From there I've gone through every Flake8 plugin in the pip freeze
package list installing it and running flake8 test.py
until the crash was reproduced.
The issue is with good_smell:
$ pip install good_smell
...
Successfully installed astor-0.7.1 astpretty-1.4.0 dataclasses-0.6 fire-0.1.3 good-smell-0.13.0
$ flake8 test.py
Traceback (most recent call last):
File "/tmp/tmp.8vn2ngI1KH/venv/bin/flake8", line 8, in <module>
sys.exit(main())
File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/flake8/main/cli.py", line 18, in main
app.run(argv)
File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/flake8/main/application.py", line 393, in run
self._run(argv)
File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/flake8/main/application.py", line 381, in _run
self.run_checks()
File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/flake8/main/application.py", line 300, in run_checks
self.file_checker_manager.run()
File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/flake8/checker.py", line 331, in run
self.run_serial()
File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/flake8/checker.py", line 315, in run_serial
checker.run_checks()
File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/flake8/checker.py", line 598, in run_checks
self.run_ast_checks()
File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/flake8/checker.py", line 502, in run_ast_checks
for (line_number, offset, text, check) in runner:
File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/flake8_aaa/checker.py", line 48, in run
self.load()
File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/flake8_aaa/checker.py", line 37, in load
self.ast_tokens = asttokens.ASTTokens(''.join(self.lines), tree=self.tree)
File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/asttokens/asttokens.py", line 65, in __init__
self.mark_tokens(self._tree)
File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/asttokens/asttokens.py", line 76, in mark_tokens
MarkTokens(self).visit_tree(root_node)
File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/asttokens/mark_tokens.py", line 49, in visit_tree
util.visit_tree(node, self._visit_before_children, self._visit_after_children)
File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/asttokens/util.py", line 199, in visit_tree
ret = postvisit(current, par_value, value)
File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/asttokens/mark_tokens.py", line 92, in _visit_after_children
nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/asttokens/mark_tokens.py", line 189, in handle_attr
name = self._code.next_token(dot)
File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/asttokens/asttokens.py", line 141, in next_token
while is_non_coding_token(self._tokens[i].type):
IndexError: list index out of range
The reason that this IndexError
happens is because good_smell manipulates the AST of the file being linted before all other plugins have completed.
Flake8-AAA receives the AST of the file being checked from Flake8 by depending on tree
as per the plugin development docs. It then passes the AST to ASTTokens to pair nodes back to the source code that generated them. The crash in Flake8-AAA is coming from ASTTokens because it can't match the AST its been passed with the contents of the file.
From my reading of the good_smell code, that's because good_smell also depends on tree
here. In this example it then manipulates the AST here https://github.com/Tadaboody/good_smell/blob/master/good_smell/smells/nested_for.py#L37-L39. This means that the file being linted no longer matches the AST in tree
by the time Flake8-AAA uses it.
I'm going to close this issue because it's not fixable in Flake8-AAA. As I've shown above, the snippet works with Python 3.6 and 3.7. Also if you install all the Flake8 plugins in your pip freeze
list except good_smell
, then it will also work fine.
good_smell should not manipulate the tree passed by Flake8 - that's a bad smell. Flake8 plugins that wish to format files should be written as a formatting plugin.
That's some serious investigation work, thanks for doing all that.
Thanks for looking into good_smell
@jamescooke, hope it wasn't too much trouble! Thought you might want to know that flake8 formatting plugins are for controlling the format in which error codes are displayed, so I'll keep the file reformatting outside of flake8.
You're right that my tool hasn't been playing nice and I'm working on fixing it!
Thanks for looking into good_smell @jamescooke, hope it wasn't too much trouble!
No problem. It was interested debugging and I learned a lot about Flake8! I had not grokked that the tree
passed to each plugin is a single object that has to be shared nicely between plugins.
Thought you might want to know that flake8 formatting plugins are for controlling the format in which error codes are displayed.
Thanks ๐ I hadn't worked that out - I thought that the write()
method was for writing newly formatted file lines back to the file, or back to Flake8 in some magic way https://flake8.pycqa.org/en/latest/plugin-development/formatters.html#flake8.formatting.base.BaseFormatter.write