Crashes in parsetab due to ply race condition

[jfly]

When multiple Python processes are simultaneously doing a from flanker.addresslib import address, it's possible for some of them to crash in ply code. I can fairly reliably reproduce this crash with the following dockerfile:

$ docker run $(docker build -q https://raw.githubusercontent.com/jfly/jfly.github.io/master/misc/ply-race/Dockerfile-race-demo)
Symbol 'mailbox_or_url_list' is unreachable
Symbol 'delim' is unreachable
Symbol 'mailbox_or_url' is unreachable
Symbol 'url' is unreachable
Symbol 'mailbox_or_url_list' is unreachable
Symbol 'delim' is unreachable
Symbol 'mailbox_or_url' is unreachable
Symbol 'url' is unreachable
Symbol 'mailbox_or_url_list' is unreachable
Symbol 'delim' is unreachable
Symbol 'mailbox_or_url' is unreachable
Symbol 'url' is unreachable
Symbol 'mailbox_or_url_list' is unreachable
Symbol 'delim' is unreachable
Symbol 'mailbox_or_url' is unreachable
Symbol 'url' is unreachable
Symbol 'mailbox_or_url_list' is unreachable
Symbol 'delim' is unreachable
Symbol 'mailbox_or_url' is unreachable
Symbol 'url' is unreachable
Symbol 'mailbox_or_url_list' is unreachable
Symbol 'delim' is unreachable
Symbol 'mailbox_or_url' is unreachable
Symbol 'mailbox' is unreachable
Symbol 'url' is unreachable
Symbol 'angle_addr' is unreachable
Symbol 'name_addr' is unreachable
Symbol 'phrase' is unreachable
Symbol 'mailbox_or_url_list' is unreachable
Process Process-5:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "importing.py", line 5, in someFunc
    from flanker.addresslib import address
  File "/usr/local/lib/python3.7/site-packages/flanker/addresslib/address.py", line 49, in <module>
    from flanker.addresslib._parser.parser import (Mailbox, Url, mailbox_parser,
  File "/usr/local/lib/python3.7/site-packages/flanker/addresslib/_parser/parser.py", line 161, in <module>
    tabmodule='mailbox_parsetab')
  File "/usr/local/lib/python3.7/site-packages/ply/yacc.py", line 3293, in yacc
    read_signature = lr.read_table(tabmodule)
  File "/usr/local/lib/python3.7/site-packages/ply/yacc.py", line 1987, in read_table
    if parsetab._tabversion != __tabversion__:
AttributeError: module 'flanker.addresslib._parser.mailbox_parsetab' has no attribute '_tabversion'
Symbol 'mailbox_or_url_list' is unreachable
Symbol 'delim' is unreachable
Symbol 'mailbox_or_url' is unreachable
Symbol 'mailbox' is unreachable
Symbol 'addr_spec' is unreachable
Symbol 'angle_addr' is unreachable
Symbol 'delim' is unreachable
Symbol 'mailbox_or_url' is unreachable
Symbol 'mailbox' is unreachable
Symbol 'name_addr' is unreachable
Symbol 'addr_spec' is unreachable
Symbol 'angle_addr' is unreachable
Symbol 'phrase' is unreachable
Symbol 'name_addr' is unreachable
Symbol 'phrase' is unreachable
Symbol 'local_part' is unreachable
Symbol 'local_part' is unreachable
Symbol 'domain' is unreachable
Symbol 'quoted_string' is unreachable
Symbol 'domain_literal' is unreachable
Symbol 'domain' is unreachable
Symbol 'quoted_string_text' is unreachable
Symbol 'domain_literal_text' is unreachable
Symbol 'quoted_string' is unreachable
Symbol 'domain_literal' is unreachable
Symbol 'quoted_string_text' is unreachable
Symbol 'domain_literal_text' is unreachable
Symbol 'mailbox_or_url_list' is unreachable
Symbol 'delim' is unreachable
Symbol 'mailbox_or_url' is unreachable
Symbol 'mailbox' is unreachable
Symbol 'addr_spec' is unreachable
Symbol 'angle_addr' is unreachable
Symbol 'name_addr' is unreachable
Symbol 'phrase' is unreachable
Symbol 'local_part' is unreachable
Symbol 'domain' is unreachable
Symbol 'quoted_string' is unreachable
Symbol 'domain_literal' is unreachable
Symbol 'quoted_string_text' is unreachable
Symbol 'mailbox_or_url_list' is unreachable
Symbol 'delim' is unreachable
Symbol 'mailbox_or_url' is unreachable
Symbol 'mailbox' is unreachable
Symbol 'addr_spec' is unreachable
Symbol 'angle_addr' is unreachable
Symbol 'name_addr' is unreachable
Symbol 'phrase' is unreachable
Symbol 'local_part' is unreachable
Symbol 'domain' is unreachable
Symbol 'quoted_string' is unreachable
Symbol 'domain_literal' is unreachable
Symbol 'quoted_string_text' is unreachable
Symbol 'domain_literal_text' is unreachable
Symbol 'mailbox_or_url_list' is unreachable
Symbol 'delim' is unreachable
Symbol 'mailbox_or_url' is unreachable
Symbol 'mailbox' is unreachable
Symbol 'addr_spec' is unreachable
Symbol 'angle_addr' is unreachable
Symbol 'name_addr' is unreachable
Symbol 'phrase' is unreachable
Symbol 'local_part' is unreachable
Symbol 'domain' is unreachable
Symbol 'quoted_string' is unreachable
Symbol 'domain_literal' is unreachable
Symbol 'quoted_string_text' is unreachable
Symbol 'domain_literal_text' is unreachable
Symbol 'domain_literal_text' is unreachable
Symbol 'mailbox_or_url_list' is unreachable
Symbol 'delim' is unreachable
Symbol 'mailbox_or_url' is unreachable
Symbol 'mailbox' is unreachable
Symbol 'addr_spec' is unreachable
Symbol 'angle_addr' is unreachable
Symbol 'name_addr' is unreachable
Symbol 'phrase' is unreachable
Symbol 'local_part' is unreachable
Symbol 'domain' is unreachable
Symbol 'quoted_string' is unreachable
Symbol 'domain_literal' is unreachable
Symbol 'quoted_string_text' is unreachable
Symbol 'domain_literal_text' is unreachable
Symbol 'mailbox_or_url_list' is unreachable
Symbol 'mailbox_or_url_list' is unreachable
Symbol 'delim' is unreachable
Symbol 'mailbox_or_url_list' is unreachable
Symbol 'delim' is unreachable
Symbol 'delim' is unreachable
Symbol 'mailbox_or_url_list' is unreachable
Symbol 'delim' is unreachable
Symbol 'mailbox_or_url_list' is unreachable
Process Process-6:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "importing.py", line 5, in someFunc
    from flanker.addresslib import address
  File "/usr/local/lib/python3.7/site-packages/flanker/addresslib/address.py", line 49, in <module>
    from flanker.addresslib._parser.parser import (Mailbox, Url, mailbox_parser,
  File "/usr/local/lib/python3.7/site-packages/flanker/addresslib/_parser/parser.py", line 171, in <module>
    tabmodule='url_parsetab')
  File "/usr/local/lib/python3.7/site-packages/ply/yacc.py", line 3293, in yacc
    read_signature = lr.read_table(tabmodule)
  File "/usr/local/lib/python3.7/site-packages/ply/yacc.py", line 1987, in read_table
    if parsetab._tabversion != __tabversion__:
AttributeError: module 'flanker.addresslib._parser.url_parsetab' has no attribute '_tabversion'
Symbol 'delim' is unreachable
Symbol 'mailbox_or_url_list' is unreachable
Symbol 'delim' is unreachable
Process Process-7:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "importing.py", line 5, in someFunc
    from flanker.addresslib import address
  File "/usr/local/lib/python3.7/site-packages/flanker/addresslib/address.py", line 49, in <module>
    from flanker.addresslib._parser.parser import (Mailbox, Url, mailbox_parser,
  File "/usr/local/lib/python3.7/site-packages/flanker/addresslib/_parser/parser.py", line 176, in <module>
    tabmodule='mailbox_or_url_parsetab')
  File "/usr/local/lib/python3.7/site-packages/ply/yacc.py", line 3293, in yacc
    read_signature = lr.read_table(tabmodule)
  File "/usr/local/lib/python3.7/site-packages/ply/yacc.py", line 1984, in read_table
    exec('import %s' % module)
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/flanker/addresslib/_parser/mailbox_or_url_parsetab.py", line 54
    ('quoted_string_text -> quoted_string_text QTEXT','quoted_string_text',2,'p_expression_quoted_string_text','parser.py',83),
                                                                                                                              ^
SyntaxError: unexpected EOF while parsing
Process Process-4:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "importing.py", line 5, in someFunc
    from flanker.addresslib import address
  File "/usr/local/lib/python3.7/site-packages/flanker/addresslib/address.py", line 49, in <module>
    from flanker.addresslib._parser.parser import (Mailbox, Url, mailbox_parser,
  File "/usr/local/lib/python3.7/site-packages/flanker/addresslib/_parser/parser.py", line 176, in <module>
    tabmodule='mailbox_or_url_parsetab')
  File "/usr/local/lib/python3.7/site-packages/ply/yacc.py", line 3293, in yacc
    read_signature = lr.read_table(tabmodule)
  File "/usr/local/lib/python3.7/site-packages/ply/yacc.py", line 1987, in read_table
    if parsetab._tabversion != __tabversion__:
AttributeError: module 'flanker.addresslib._parser.mailbox_or_url_parsetab' has no attribute '_tabversion'
<module 'flanker.addresslib.address' from '/usr/local/lib/python3.7/site-packages/flanker/addresslib/address.py'>
<module 'flanker.addresslib.address' from '/usr/local/lib/python3.7/site-packages/flanker/addresslib/address.py'>
<module 'flanker.addresslib.address' from '/usr/local/lib/python3.7/site-packages/flanker/addresslib/address.py'>
<module 'flanker.addresslib.address' from '/usr/local/lib/python3.7/site-packages/flanker/addresslib/address.py'>
<module 'flanker.addresslib.address' from '/usr/local/lib/python3.7/site-packages/flanker/addresslib/address.py'>
<module 'flanker.addresslib.address' from '/usr/local/lib/python3.7/site-packages/flanker/addresslib/address.py'>

This bug has been reported before in #168, and PR #188 attempted to work around the issue. However, the workaround in that PR doesn't help if you have a different version of ply installed than flanker used to generate its parsetab files. ply will notice that the parsetab files were generated with a different version of flanker, and decide to regenerate the files, causing the same bug. As I understand it, there are two ways flanker could work around this issue:

1. Update setup.py to pin ply to exactly version 3.10 (this is the version used in #188). This was actually discussed here: #188 (comment), but rejected (I don't think the people involved in that discussion were aware of the race condition).
2. Change the invocations of yacc to pass write_tables=False, as @jlev did in spacedogXYZ@54d5008.

However, I would argue that neither of these are great solutions, and they are just working around a fundamental race condition in ply. I've sent in a PR to ply to try to fix the root bug (dabeaz/ply#184), but I wanted to file this issue with you guys just so you're aware of it. Currently, it's pretty unfortunate that someone who does pip install flanker will automatically end up with ply == 3.11, and be exposed to this race condition.

[jfly]

My PR to ply was rejected: dabeaz/ply#184. How do you feel about implementing one of the 2 workarounds/fixes detailed in my original post?