Plethora of 2.7 parse issues

[ndevenish]

Tried on our large and some-parts-somewhat-ancient 2.7-compatible codebase (the kind it'd be nice to have, say, automatic transformations for...). Most of the files worked but a few recurring issues - perhaps these come under "some features not supported" but I didn't see an explicit list anywhere. All come from real files that work.

• Complex numbers don't work

1j
# AnnotationError: no ordering relation is defined for complex numbers

• Various issues with tuple parameter unpacking:

def a((b,c), d):
  pass
# AnnotationError: Expected ',' but found ')'
def some((a,b,c)):
  pass
# AnnotationError: Expected ':' but found ')'

• Slicing with steps:

some_sliceable[::]
# AnnotationError: Expected 'None' but found ']'

• Calling functions with extra newlines:

def call_func(a, something_else): # valid function
  pass
call_func((1
,), something_else=True)

# AnnotationError: Expected 'something_else' but found ')'

• <> operator:

True <> False
# AnnotationError: Expected '!=' but found '<>'

• BOM at start of file:

pasta.parse("\xef\xbb\xbf\r\nimport sys")
# Expected 'import' but found '\xef'`

• Old-style repr (for obvious reasons having trouble typing this one in markdown)

pasta.parse("\x60True\x60")
# Expected 'repr' but found '\x60'

• Something with exec and lines?

exec("", None, None)
True
# Expected 'True' but found ')'

[soupytwist]

Thanks for the detailed breakdown. Indeed, these are mostly things that I wasn't aware of / have been overlooked. I appreciate you taking the time to list them out so succinctly. Most of these should be simple to fix now that I know what to look for.

[soupytwist]

Pushed a few fixes in f995fc5, 8e07293 and 91cbb35. More to come...

[soupytwist]

Tuple parameter unpacking should be fixed in 1d7d268. Still a few more.

[soupytwist]

The weird case with newlines in function calls is fixed in 3ce4123. This is actually because of the newline before trailing comma in the tuple.

These are remaining:

• BOM at start of file:

pasta.parse("\xef\xbb\xbf\r\nimport sys")
# Expected 'import' but found '\xef'`

• Old-style repr (for obvious reasons having trouble typing this one in markdown)

pasta.parse("\x60True\x60")
# Expected 'repr' but found '\x60'

• Something with exec and lines?

exec("", None, None)
True
# Expected 'True' but found ')'

[soupytwist]

I believe these are all fixed now except for the BOM parsing.