psf/black

Cannot parse multiline f-string containing multiline string

tarper24 opened this issue · 1 comments

tarper24 commented

Describe the bug

Black fails to parse a file containing a multiline f-string which contains a multiline string within curly braces

To Reproduce

I created this MRC

f"""{'''
'''}"""

And just run black on the file:

$ black file.py

The resulting error is:

error: cannot format file.py: Cannot parse: 1:5: f"""{'''

Expected behavior

Successfully parse as it did in version <= 24.4.0

Environment

  • Black's version: 24.4.2
  • OS and Python version: Windows 11, Python 3.10, 3.11, 3.12

Additional context

Related to #4329

JelleZijlstra commented 
% python3 -m blib2to3.pgen2.tokenize multil.py
1,0-1,4:	FSTRING_START	'f"""'
1,4-1,4:	FSTRING_MIDDLE	''
1,4-1,5:	LBRACE	'{'
1,5-2,0:	FSTRING_MIDDLE	"'''\n"
2,0-2,3:	FSTRING_END	"'''"
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/jelle/py/black/src/blib2to3/pgen2/tokenize.py", line 1115, in <module>
    tokenize(open(sys.argv[1]).readline)
  File "/Users/jelle/py/black/src/blib2to3/pgen2/tokenize.py", line 281, in tokenize
    tokenize_loop(readline, tokeneater)
  File "/Users/jelle/py/black/src/blib2to3/pgen2/tokenize.py", line 288, in tokenize_loop
    for token_info in generate_tokens(readline):
  File "/Users/jelle/py/black/src/blib2to3/pgen2/tokenize.py", line 680, in generate_tokens
    fstring_state.leave_fstring()
  File "/Users/jelle/py/black/src/blib2to3/pgen2/tokenize.py", line 541, in leave_fstring
    assert state == STATE_MIDDLE
AssertionError

It incorrectly emits FSTRING_MIDDLE even though the inner string is inside braces and isn't an f-string.

cc @tusharsadhwani

  • 👀1