Some sort of race condition in packaging.requirement
alex opened this issue · 6 comments
import sys
import threading
from packaging.requirements import Requirement
N_THREADS = 32
def target():
Requirement("x[]")
def main(argv):
threads = []
for i in range(N_THREADS):
threads.append(threading.Thread(target=target))
for t in threads:
t.start()
for t in threads:
t.join()
print("Ok")
if __name__ == "__main__":
main(sys.argv)
This fails every once in a while. Put in a loop in your terminal for best results.
Oh, where are my manners, here's the traceback:
Exception in thread Thread-3:
Traceback (most recent call last):
File "/Users/alex_gaynor/.pyenv/versions/2.7.13/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/Users/alex_gaynor/.pyenv/versions/2.7.13/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "t.py", line 10, in target
Requirement("x[]")
File "/Users/alex_gaynor/.virtualenvs/tempenv-420c1556839f5/lib/python2.7/site-packages/packaging/requirements.py", line 94, in __init__
requirement_string[e.loc:e.loc + 8]))
InvalidRequirement: Invalid requirement, parse error at "''"
Alex - many thanks for closing this -- I was able to use your code to replicate and test that this is indeed fixed. One followup question -- will this be part of an official release soon (so that I can get it with pip upgrade instead of doing a manual compile/install?) Thanks again!
This is still biting me with packaging==21.3. I understand that the PR (#105) tried two fixes (and went with the second):
- Wrapping the parsing in a lock. I tested, and this actually works for me.
- Initializing the pyparsing engine at import time, but no synchronization at parse time. I still get pyparsing errors when I run this inside a lot of threads. Indeed, the pyparsing folks recommend never getting inside pyparsing's codebase from more than one thread at the same time (pyparsing/pyparsing#89 (comment)).
I also understand that packaging 22.0 will have a tailor-made parser instead of depending on pyparsing. Will this parser have global state which must be protected by a lock, or can I assume that calling packaging.requirements.Requirement
will be thread-safe?
Thanks!
We no longer use pyparsing in main
, so those race conditions aren't an issue.
I'll assume the race conditions are no longer an issue? Thanks!
Indeed, typo fixed. :)