jamescooke/flake8-aaa

IndexError

Dreamsorcerer opened this issue · 7 comments

Dreamsorcerer commented

With this file:

for i in seq_a:
    for j in seq_b:
        print(i, j)

I get this output:

Traceback (most recent call last):
  File "/home/ubuntu/.local/bin/flake8", line 11, in <module>
    sys.exit(main())
  File "/home/ubuntu/.local/lib/python3.6/site-packages/flake8/main/cli.py", line 18, in main
    app.run(argv)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/flake8/main/application.py", line 393, in run
    self._run(argv)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/flake8/main/application.py", line 381, in _run
    self.run_checks()
  File "/home/ubuntu/.local/lib/python3.6/site-packages/flake8/main/application.py", line 300, in run_checks
    self.file_checker_manager.run()
  File "/home/ubuntu/.local/lib/python3.6/site-packages/flake8/checker.py", line 331, in run
    self.run_serial()
  File "/home/ubuntu/.local/lib/python3.6/site-packages/flake8/checker.py", line 315, in run_serial
    checker.run_checks()
  File "/home/ubuntu/.local/lib/python3.6/site-packages/flake8/checker.py", line 598, in run_checks
    self.run_ast_checks()
  File "/home/ubuntu/.local/lib/python3.6/site-packages/flake8/checker.py", line 502, in run_ast_checks
    for (line_number, offset, text, check) in runner:
  File "/home/ubuntu/.local/lib/python3.6/site-packages/flake8_aaa/checker.py", line 48, in run
    self.load()
  File "/home/ubuntu/.local/lib/python3.6/site-packages/flake8_aaa/checker.py", line 37, in load
    self.ast_tokens = asttokens.ASTTokens(''.join(self.lines), tree=self.tree)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/asttokens/asttokens.py", line 65, in __init__
    self.mark_tokens(self._tree)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/asttokens/asttokens.py", line 76, in mark_tokens
    MarkTokens(self).visit_tree(root_node)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/asttokens/mark_tokens.py", line 49, in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/asttokens/util.py", line 199, in visit_tree
    ret = postvisit(current, par_value, value)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/asttokens/mark_tokens.py", line 92, in _visit_after_children
    nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/asttokens/mark_tokens.py", line 189, in handle_attr
    name = self._code.next_token(dot)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/asttokens/asttokens.py", line 141, in next_token
    while is_non_coding_token(self._tokens[i].type):
IndexError: list index out of range
jamescooke commented

Hi @Dreamsorcerer - thanks for opening this issue.

Looking at the last line of the stack trace, it seems like this is an error in asttokens (if you're interested, that's this library that we use to help analyse the Python AST https://github.com/gristlabs/asttokens)...

  File "/home/ubuntu/.local/lib/python3.6/site-packages/asttokens/asttokens.py", line 141, in next_token
    while is_non_coding_token(self._tokens[i].type):

Could you help me by confirming:

  • You're on Python 3.6 - is that right? Could you give the output of python --version?
  • Could you share the versions of Flake8, Flake8-AAA and asttokens that are installed? Doing pip freeze and sharing the output would be helpful.

Thanks 👍

jamescooke commented

I'm not currently able to reproduce with Python 3.7 and Flake8-AAA v0.9.0:

$ pip install flake8-aaa flake8
Collecting flake8-aaa
  Using cached flake8_aaa-0.9.0-py3-none-any.whl (16 kB)
Collecting flake8
  Using cached flake8-3.7.9-py2.py3-none-any.whl (69 kB)
Collecting asttokens>=2
  Downloading asttokens-2.0.4-py2.py3-none-any.whl (20 kB)
Collecting entrypoints<0.4.0,>=0.3.0
  Using cached entrypoints-0.3-py2.py3-none-any.whl (11 kB)
Collecting mccabe<0.7.0,>=0.6.0
  Using cached mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB)
Collecting pyflakes<2.2.0,>=2.1.0
  Using cached pyflakes-2.1.1-py2.py3-none-any.whl (59 kB)
Collecting pycodestyle<2.6.0,>=2.5.0
  Using cached pycodestyle-2.5.0-py2.py3-none-any.whl (51 kB)
Collecting six
  Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Installing collected packages: six, asttokens, flake8-aaa, entrypoints, mccabe, pyflakes, pycodestyle, flake8
Successfully installed asttokens-2.0.4 entrypoints-0.3 flake8-3.7.9 flake8-aaa-0.9.0 mccabe-0.6.1 pycodestyle-2.5.0 pyflakes-2.1.1 six-1.14.0
$ cat > test.py
for i in seq_a:
    for j in seq_b:
        print(i, j)
$ flake8 test.py 
test.py:1:10: F821 undefined name 'seq_a'
test.py:2:14: F821 undefined name 'seq_b'

Instead, Flake8 just errors that seq_a and seq_b are not defined. Hopefully the answers to the questions above will help me reproduce. 🤞

Dreamsorcerer commented

This is after uninstalling flake8-aaa, in case you wonder why it's not in there.

> python3 --version
Python 3.6.8
> flake8 --version
3.7.9 (assertive: 1.2.1, flake8-annotations: 2.1.0, flake8-annotations-complexity: 0.0.4, flake8-bandit: 2.1.2, flake8-broken-line: 0.2.0, flake8-bugbear: 20.1.4, flake8-class-attributes-order: 0.1.0, flake8-comprehensions: 3.2.2, flake8-darglint: 0.4.1, flake8-debugger: 3.2.1, flake8-docstrings: 1.5.0, pydocstyle: 5.0.2, flake8-eradicate: 0.3.0, flake8-executable: 2.0.3, flake8-requirements: 1.3.1, flake8-sorted-keys: 0.2.0, flake8-string-format: 0.3.0, flake8-todos: 1.0.0, flake8.datetimez: 19.5.4.0, flake8_commas: 2.0.0, flake8_isort: 2.9.1, flake8_quotes: 3.0.0, good-smell: 0.1, import-order: 0.18.1, logging-format: 0.6.0, mccabe: 0.6.1, naming: 0.10.0, pycodestyle: 2.5.0, pyflakes: 2.1.1, radon: 4.1.0, rst-docstrings: 0.0.12, wemake_python_styleguide: 0.14.0) CPython 3.6.8 on Linux
> pip3 freeze
acme==0.26.1
Aggregated-Notifications-API==1.0
aiohttp==3.6.2
aiohttp-debugtoolbar==0.6.0
aiohttp-devtools==0.13.1
aiohttp-jinja2==1.2.0
aiohttp-security==0.4.0
aiohttp-session==2.9.0
asn1crypto==0.24.0
astor==0.8.1
astpretty==2.0.0
asttokens==2.0.4
async-timeout==3.0.1
asyncio-glib==0.1
attrs==19.3.0
bandit==1.6.2
beautifulsoup4==4.6.3
certbot==0.26.0.dev0
certbot-nginx==0.23.0
certifi==2019.11.28
cffi==1.14.0
chardet==3.0.4
click==7.1.2
clickclick==1.2.2
cognitive-complexity==0.0.4
colorama==0.4.3
ConfigArgParse==0.11.0
configobj==5.0.6
connexion==1.5.3
contractions==0.0.17
cryptography==2.9.2
Cython==0.29.14
darglint==1.2.3
dataclasses==0.6
devtools==0.5.1
distro-info===0.18ubuntu0.18.04.1
docutils==0.16
entrypoints==0.3
eradicate==1.0
fire==0.1.3
flake8==3.7.9
flake8-annotations==2.1.0
flake8-annotations-complexity==0.0.4
flake8-assertive==1.2.1
flake8-bandit==2.1.2
flake8-broken-line==0.2.0
flake8-bugbear==20.1.4
flake8-class-attributes-order==0.1.0
flake8-commas==2.0.0
flake8-comprehensions==3.2.2
flake8-datetimez==19.5.4.0
flake8-debugger==3.2.1
flake8-docstrings==1.5.0
flake8-eradicate==0.3.0
flake8-executable==2.0.3
flake8-import-order==0.18.1
flake8-isort==2.9.1
flake8-logging-format==0.6.0
flake8-plugin-utils==1.0.0
flake8-polyfill==1.0.2
flake8-quotes==3.0.0
flake8-requirements==1.3.1
flake8-rst-docstrings==0.0.12
flake8-sorted-keys==0.2.0
flake8-string-format==0.3.0
flake8-todos==0.1.3
Flask==1.0.2
future==0.18.2
gbulb==0.6.1
gitdb==4.0.4
gitdb2==2.0.6
GitPython==3.1.1
good-smell==0.13.0
idna==2.9
idna-ssl==1.1.0
importlib-metadata==1.6.0
inflect==2.1.0
inflection==0.3.1
isort==4.3.21
itsdangerous==0.24
Jinja2==2.11.2
josepy==1.1.0
jsonschema==2.6.0
keyring==10.6.0
keyrings.alt==3.0
lz4==3.0.2
Mako==1.0.7
mando==0.6.4
MarkupSafe==1.1.1
mccabe==0.6.1
mock==2.0.0
more-itertools==8.1.0
multidict==4.7.5
mypy==0.770
mypy-extensions==0.4.3
netifaces==0.10.4
nltk==3.4
outcome==1.0.1
parsedatetime==2.4
pbr==5.4.5
pep8-naming==0.10.0
pluggy==0.6.0
py==1.5.2
pycairo==1.18.2
pycodestyle==2.5.0
pycparser==2.20
pycrypto==2.6.1
pydocstyle==5.0.2
pyflakes==2.1.1
Pygments==2.6.1
PyGObject==3.34.0
pynput==1.4.2
pyOpenSSL==17.5.0
pyparsing==2.2.0
pyRFC3339==1.0
python-apt==1.6.5+ubuntu0.2
python-debian==0.1.32
python-xlib==0.25
pytz==2018.3
pyxdg==0.25
PyYAML==5.3.1
radon==4.1.0
Represent==1.6.0
requests==2.22.0
requests-toolbelt==0.8.0
restructuredtext-lint==1.3.0
SecretStorage==2.3.1
singledispatch==3.4.0.3
six==1.14.0
smmap==3.0.2
smmap2==2.0.5
snowballstemmer==2.0.0
SQLAlchemy==1.3.16
sqlalchemy-aio==0.15.0
sqlparse==0.3.1
stevedore==1.32.0
swagger-spec-validator==2.4.1
testfixtures==6.14.1
toml==0.10.0
tornado==5.1.1
tox==2.5.0
trafaret==2.0.2
typed-ast==1.4.1
typing-extensions==3.7.4.2
unattended-upgrades==0.1
urllib3==1.25.7
virtualenv==15.1.0
watchgod==0.6
wemake-python-styleguide==0.14.0
Werkzeug==0.14.1
yarl==1.4.2
zipp==3.1.0
zope.component==4.3.0
zope.event==4.2.0
zope.hookable==4.0.4
zope.interface==4.3.2
jamescooke commented

Thanks for providing that pip freeze output - you've got a lot of packages installed!

Firstly, the good news is that Python 3.6 works with the example snippet:

$ python --version
Python 3.6.7
$ pip install flake8 flake8-aaa
...
Successfully installed asttokens-2.0.4 entrypoints-0.3 flake8-3.7.9 flake8-aaa-0.9.0 mccabe-0.6.1 pycodestyle-2.5.0 pyflakes-2.1.1 six-1.14.0
$ cat > test.py
for i in seq_a:
    for j in seq_b:
        print(i, j)
$ flake8 test.py
test.py:1:10: F821 undefined name 'seq_a'
test.py:2:14: F821 undefined name 'seq_b'

From there I've gone through every Flake8 plugin in the pip freeze package list installing it and running flake8 test.py until the crash was reproduced.

The issue is with good_smell:

$ pip install good_smell
...
Successfully installed astor-0.7.1 astpretty-1.4.0 dataclasses-0.6 fire-0.1.3 good-smell-0.13.0
$ flake8 test.py
Traceback (most recent call last):
  File "/tmp/tmp.8vn2ngI1KH/venv/bin/flake8", line 8, in <module>
    sys.exit(main())
  File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/flake8/main/cli.py", line 18, in main
    app.run(argv)
  File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/flake8/main/application.py", line 393, in run
    self._run(argv)
  File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/flake8/main/application.py", line 381, in _run
    self.run_checks()
  File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/flake8/main/application.py", line 300, in run_checks
    self.file_checker_manager.run()
  File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/flake8/checker.py", line 331, in run
    self.run_serial()
  File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/flake8/checker.py", line 315, in run_serial
    checker.run_checks()
  File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/flake8/checker.py", line 598, in run_checks
    self.run_ast_checks()
  File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/flake8/checker.py", line 502, in run_ast_checks
    for (line_number, offset, text, check) in runner:
  File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/flake8_aaa/checker.py", line 48, in run
    self.load()
  File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/flake8_aaa/checker.py", line 37, in load
    self.ast_tokens = asttokens.ASTTokens(''.join(self.lines), tree=self.tree)
  File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/asttokens/asttokens.py", line 65, in __init__
    self.mark_tokens(self._tree)
  File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/asttokens/asttokens.py", line 76, in mark_tokens
    MarkTokens(self).visit_tree(root_node)
  File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/asttokens/mark_tokens.py", line 49, in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
  File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/asttokens/util.py", line 199, in visit_tree
    ret = postvisit(current, par_value, value)
  File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/asttokens/mark_tokens.py", line 92, in _visit_after_children
    nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
  File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/asttokens/mark_tokens.py", line 189, in handle_attr
    name = self._code.next_token(dot)
  File "/tmp/tmp.8vn2ngI1KH/venv/lib/python3.6/site-packages/asttokens/asttokens.py", line 141, in next_token
    while is_non_coding_token(self._tokens[i].type):
IndexError: list index out of range

The reason that this IndexError happens is because good_smell manipulates the AST of the file being linted before all other plugins have completed.

Flake8-AAA receives the AST of the file being checked from Flake8 by depending on tree as per the plugin development docs. It then passes the AST to ASTTokens to pair nodes back to the source code that generated them. The crash in Flake8-AAA is coming from ASTTokens because it can't match the AST its been passed with the contents of the file.

From my reading of the good_smell code, that's because good_smell also depends on tree here. In this example it then manipulates the AST here https://github.com/Tadaboody/good_smell/blob/master/good_smell/smells/nested_for.py#L37-L39. This means that the file being linted no longer matches the AST in tree by the time Flake8-AAA uses it.

I'm going to close this issue because it's not fixable in Flake8-AAA. As I've shown above, the snippet works with Python 3.6 and 3.7. Also if you install all the Flake8 plugins in your pip freeze list except good_smell, then it will also work fine.

good_smell should not manipulate the tree passed by Flake8 - that's a bad smell. Flake8 plugins that wish to format files should be written as a formatting plugin.

Dreamsorcerer commented

That's some serious investigation work, thanks for doing all that.

  • 👍2
Tadaboody commented

Thanks for looking into good_smell @jamescooke, hope it wasn't too much trouble! Thought you might want to know that flake8 formatting plugins are for controlling the format in which error codes are displayed, so I'll keep the file reformatting outside of flake8.
You're right that my tool hasn't been playing nice and I'm working on fixing it!

  • 👍1
jamescooke commented

Thanks for looking into good_smell @jamescooke, hope it wasn't too much trouble!

No problem. It was interested debugging and I learned a lot about Flake8! I had not grokked that the tree passed to each plugin is a single object that has to be shared nicely between plugins.

Thought you might want to know that flake8 formatting plugins are for controlling the format in which error codes are displayed.

Thanks 👍 I hadn't worked that out - I thought that the write() method was for writing newly formatted file lines back to the file, or back to Flake8 in some magic way https://flake8.pycqa.org/en/latest/plugin-development/formatters.html#flake8.formatting.base.BaseFormatter.write

  • 😄1