Failed to recognize comma on exec function on python2.7

[williamjamir]

Hi,

The parser does not recognize the comma on exec function with python2.7

I did try to create a PR solving this issue, I was doing a function " in_or_comma() " but I realized that the problem would be deeper than this because the visit_exec is treating the exec as a statement instead of function. Am I right?

Here it's a small reproducible example of the error:

File:
exec('x = 10' , {} )

Script:

import pasta
from pasta.augment import rename

path = r'test.py'

with open(path) as file:
        tree = pasta.parse(file.read())

Error:

pasta.base.annotate.AnnotationError: Expected 'in' but found ','
line 1: exec ('x = 10', {})

cc @nicoddemus

[soupytwist]

Hmm. So the parsed AST from this is:
"Module(body=[Exec(body=Tuple(elts=[Str(s='x = 10'), Dict(keys=[], values=[])], ctx=Load()), globals=None, locals=None)])"

I can't reproduce it on python 2.7.6:

import pasta
t = pasta.parse("exec('x = 10' , {} )")

It shouldn't be expecting "in" unless the ast parses out something for globals. What's your python version? Can you provide the output of:

import ast
ast.dump(ast.parse("exec('x = 10' , {} )"))

[williamjamir]

I'm using Python 2.7.14

Below you can check the output requested

$ python
Python 2.7.14 | packaged by conda-forge | (default, Dec 25 2017, 01:16:05) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ast
>>> ast.dump(ast.parse("exec('x = 10' , {} )"))
"Module(body=[Exec(body=Str(s='x = 10'), globals=Dict(keys=[], values=[]), locals=None)])"

[soupytwist]

Okay, that explains it anyway :)

Looks like some change in the parser, so I'll have to account for both. Thanks for reporting it!

[williamjamir]

Just out of curiosity (I know it will not interfere with this problem), this different behavior occurred in the transition from Python 2.7.8 to 2.7.9

Checking the changelog I found this:

• Issue #21591: Correctly handle qualified exec statements in tuple form by moving compatibility layer from run-time to AST transformation.

[soupytwist]

It seems I was too hasty.

[williamjamir]

Executing the code available on the master I found this issue with exec too.

Executing against a single line with exec it's ok:
exec('cmd', {})

But when I added something after the exec, the parse breaks =/

exec('cmd', {})
print()

Error:

pasta.base.annotate.AnnotationError: Expected 'print' but found ')'
line 1: exec('cmd', {})

Full Stack

Traceback (most recent call last):                                                        
  File "script.py", line 6, in <module>                                                       
    tree = pasta.parse(file.read())                                                       
  File "w:\william\repo_pasta\pasta\__init__.py", line 25, in parse                       
    annotator.visit(t)                                                                    
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 1048, in visit                
    super(AstAnnotator, self).visit(node)                                                 
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 115, in visit                 
    super(BaseVisitor, self).visit(node)                                                  
  File "W:\alfasim\envs\test_pasta\lib\ast.py", line 241, in visit                        
    return visitor(node)                                                                  
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 79, in wrapped                
    f(self, node, *args, **kwargs)                                                        
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 168, in visit_Module          
    self.generic_visit(node)                                                              
  File "W:\alfasim\envs\test_pasta\lib\ast.py", line 249, in generic_visit                
    self.visit(item)                                                                      
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 1048, in visit                
    super(AstAnnotator, self).visit(node)                                                 
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 115, in visit                 
    super(BaseVisitor, self).visit(node)                                                  
  File "W:\alfasim\envs\test_pasta\lib\ast.py", line 241, in visit                        
    return visitor(node)                                                                  
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 43, in wrapped                
    f(self, node, *args, **kwargs)                                                        
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 575, in visit_Print           
    self.attr(node, 'print_open', ['print', self.ws], default='print ')                   
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 1173, in attr                 
    attr_parts.append(self.token(attr_val))                                               
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 1106, in token                
    token_val, token.src, token.start[0], token.line))                                    
pasta.base.annotate.AnnotationError: Expected 'print' but found ')'                       
line 1: exec('cmd', {})    

[soupytwist]

Yes, I see the problem. I'll need a more generic solution for this, it might also affect a few other places.

Generally, there needs to be support for when multiple children of a node can be parenthesized. Thanks for following up on it. I'll start working on this when I get a chance.