Datastore: difficult to decipher gRPC stack trace
andrewpollock opened this issue · 0 comments
andrewpollock commented
Hi,
We've seen some sort of failure in OSV.dev's API backend (in Cloud Run), where the instance's ability to make further Datastore queries was permanently impacted (we had to kill and restart it).
Any assistance on how to further troubleshoot what happened here would be appreciated.
Environment details
- Cloud Run
- Python version and virtual environment information: Python 3.11, per https://github.com/google/osv.dev/blob/master/gcp/api/Dockerfile
- google-cloud-ndb version: 2.1.1
Steps to reproduce
Unfortunately due to not logging the queries, we're unsure of the exact query that triggered this.
Code example
specifically
or
Stack trace
Exception in thread Thread-11366361 (_run):
Traceback (most recent call last): File "/usr/local/lib/python3.11/threading.py", line 1038, in _bootstrap_inner self.run() File "/usr/local/lib/python3.11/threading.py", line 975, in run self._target(*self._args, **self._kwargs) File "src/python/grpcio/grpc/_cython/_cygrpc/thread.pyx.pxi", line 53, in grpc._cython.cygrpc._run_with_context._run File "/usr/local/lib/python3.11/site-packages/grpc/_channel.py", line 1392, in channel_spin
event = state.channel.next_call_event()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 489, in grpc._cython.cygrpc.Channel.next_call_event
File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 171, in grpc._cython.cygrpc._next_call_event
File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 172, in grpc._cython.cygrpc._next_call_event
File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 481, in grpc._cython.cygrpc.Channel.next_call_event.on_success
File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 263, in grpc._cython.cygrpc._process_integrated_call_tag
KeyError: <grpc._cython.cygrpc._BatchOperationTag object at 0x3eb06004e6b0>