Tricky SyntaxErrors

[MostAwesomeDude]

These are some of my treasures, discovered while implementing a Python parser years ago.

First, the "surprising comma."

Codepad: http://codepad.org/JiFQi7vr

def f(**args, **kwargs):
 print args, kwargs

def args = 5, 7
f(42, *args,)

Technically only f(*args,) or f(**kwargs,) is needed to tickle this one.

Second, the "not knot."

Codepad: http://codepad.org/IA7k5jEg

x = True
y = False

print not x == y
print x == not y

Again, a smaller test case, like x == not y, will provoke it.

[satwikkansal]

Hey @MostAwesomeDude I really liked your "knot not" treasure, but not sure if I understood correctly the "surprising comma" one. As far as I understand, "having a trailing comma after *args or *kwargs is not a valid syntax". But I guess you're trying to show something more than that in that example which I'm not able to catch. Could you please explain 😅

[satwikkansal]

I kinda prepared draft for the "not knot" one

not knot.

Originally suggested by @MostAwesomeDude in this issue.

x = True
y = False

Output:

>>> not x == y
True
>>> x == not y
  File "<input>", line 1
    x == not y
           ^
SyntaxError: invalid syntax

💡 Explanation:

• Operator precedence affects how an expression is evaluated, and == operator has higher precedence than not operator in Python.
• So not x == y is equivalent to not (x == y) which is equivalent to not (True == False) finally evaluating to True.
• But x == not y raises a SyntaxError because it can be thought of being equivalent to (x == not) y and not x == (not y) which you might have expected at first sight.

Let me know, if it needs any corrections

[MostAwesomeDude]

Hi!

The surprising comma is surprising because, normally, these sorts of commas are not a problem. In two similar situations, the trailing commas are legal:

f(x, y,)

f(k=v, z=w,)

And one might expect that a trailing comma is always legal in argument lists. However, due to the way that the Python syntax is designed, the argument list is defined partially with leading commas and partially with trailing commas. This conflict causes situations where a comma is trapped in the middle, and no rule will accept it.

The not knot analysis is quite good, although it misses a critical detail: the reason why not tickles this is because the precedence level in the parser containing == is the same level containing the not in operator, and the parser isn't quite able to understand that not can start an expression instead here. (To be fair to Python, this is tricky and I'm not sure how I'd fix it without redesigning the entire expression parser; it is surprising but not a bug.)

[satwikkansal]

I see, a final question, would it be correct to say that "parser expected the not to be part of not in operator (because both == and not in had same precedence), and after not being able to find the following in token after not, it raised a SyntaxError.

[MostAwesomeDude]

I worry that it glosses over how parsers work, but OTOH I don't really expect folks to understand how parsers work, so sure, that's close enough.

[satwikkansal]

Hey, I've added the Not Knot! example in the commit 6abfb50 🎉 Please feel free to suggest changes (if any).

And here's a draft I prepared for the "The surprising comma" one.

The surprising comma

Suggested by @MostAwesomeDude in this issue.

Output:

>>> def f(x, y,):
...     print(x, y)
... 
>>> def g(x=4, y=5,):
...     print(x, y)
... 
>>> def h(x, **kwargs,):
  File "<stdin>", line 1
    def h(x, **kwargs,):
                     ^
SyntaxError: invalid syntax
>>> def h(*args,):
  File "<stdin>", line 1
    def h(*args,):
                ^
SyntaxError: invalid syntax

💡 Explanation:

• Trailing comma is not always legal in formal parameters list of a Python function.
• In Python, the argument list is defined partially with leading commas and partially with trailing commas. This conflict causes situations where a comma is trapped in the middle, and no rule accepts it.

Would like to know if you've ideas to present it in more interesting way or if the explanation is insufficient.

[uranusjr]

Probably should mention the trailing comma problem is fixed in Python 3.6. The linked issue also contains some detail explanation why this happens (grammar definition in the parser).

[satwikkansal]

Thank you so much @MostAwesomeDude for the examples from your "treasure". I've added both of them in the commits be98d88 and 6abfb50. 🎉 Please feel free to reopen the issue if you think the explanations/snippets are insufficient or inaccurate.