'Generator already executing'
remko opened this issue · 2 comments
remko commented
We are using Google Cloud NDB 1.10.1 on an App Engine Standard Python3 app.
We occasionally get the error below:
with ndbClient.context():
File "/opt/python3.8/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/ndb/client.py", line 185, in context
context = context_module.Context(
File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/ndb/context.py", line 290, in __new__
id = next(_context_ids)
ValueError: generator already executing
The frequency depends on the traffic we get, but to give you an absolute number, it happens multiple times per hour.
The call to context() happens in WSGI middleware before it hits our handlers.
Any clue what might be going wrong?
remko commented
Looking at the code, it seems that the generator for context IDs is not thread-safe.
Here's a reproducer:
from google.cloud import ndb
import threading
c = ndb.Client()
def f():
for i in range(10000):
with c.context():
pass
thread1 = threading.Thread(target=f)
thread2 = threading.Thread(target=f)
thread1.start()
thread2.start()
thread1.join()
thread2.join()
chrisrossi commented
This strikes me as probably the correct diagnosis. Thank you for bringing it to my attention. I'll take a look!