500 Internal Server Error after collecting few events from pubsublite subscription

[rnm-patel]

Environment details

• OS type and version: Tried with both Windows 10 & CentOS
• Python version: 3.7.10
• pip version: 21.1.3
• google-cloud-pubsublite version: 0.5.0
• Number of partition in Topic: 1

Steps to reproduce

1. Install the latest google-cloud-pubsublite in any OS with python version >= 3.6
2. Download & run "Receive Message" example with/without flow controll settings - Link
3. After starting data collection, it will work for around 5 min, after that, it will stop suddenly and will throw "503 - Internal server error shown in below traceback. And it will continuously try to generate a new token every 5 min and eventually fails with 503 status code error.

from google.cloud.pubsublite.cloudpubsub import SubscriberClient
from google.cloud.pubsublite.types import (
    CloudRegion,
    CloudZone,
    SubscriptionPath,
    DISABLED_FLOW_CONTROL,
)

project_number = 1122334455
cloud_region = "us-central1"
zone_id = "a"
subscription_id = "your-subscription-id"

location = CloudZone(CloudRegion(cloud_region), zone_id)
subscription_path = SubscriptionPath(project_number, location, subscription_id)

with SubscriberClient() as subscriber_client:
    streaming_pull_future = subscriber_client.subscribe(
        subscription_path,
        callback=callback,
        per_partition_flow_control_settings=DISABLED_FLOW_CONTROL,
    )

    streaming_pull_future.result()

Stack trace

google.api_core.exceptions.InternalServerError: 500 Internal error encountered.
ERROR pid=30765 tid=Thread-14 file=base_events.py:default_exception_handler:1619 | Task exception was never retrieved
future: <Task finished coro=<_StreamRequestMixin._consume_request_iterator() done, defined at /root/workspace/MyApp/MyApp/bin/lib/grpc/aio/_call.py:396> exception=InternalServerError('Internal error encountered.')>
Traceback (most recent call last):
  File "/root/workspace/MyApp/MyApp/bin/lib/google/api_core/grpc_helpers_async.py", line 108, in _wrapped_aiter
    async for response in self._call:  # pragma: no branch
  File "/root/workspace/MyApp/MyApp/bin/lib/grpc/aio/_call.py", line 321, in _fetch_stream_responses
    await self._raise_for_status()
  File "/root/workspace/MyApp/MyApp/bin/lib/grpc/aio/_call.py", line 232, in _raise_for_status
    self._cython_call.status())
grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
    status = StatusCode.INTERNAL
    details = "Internal error encountered."
    debug_error_string = "{"created":"@1624279060.222179353","description":"Error received from peer ipv4:142.250.192.10:443","file":"src/core/lib/surface/call.cc","file_line":1066,"grpc_message":"Internal error encountered.","grpc_status":13}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/workspace/MyApp/MyApp/bin/lib/google/cloud/pubsublite/internal/wire/retrying_connection.py", line 100, in _run_loop
    await self._loop_connection(connection)
  File "/root/workspace/MyApp/MyApp/bin/lib/google/cloud/pubsublite/internal/wire/retrying_connection.py", line 133, in _loop_connection
    await self._read_queue.put(await read_task)
  File 
  File "/root/workspace/MyApp/MyApp/bin/lib/google/api_core/grpc_helpers_async.py", line 111, in _wrapped_aiter
    raise exceptions.from_grpc_error(rpc_error) from rpc_error
google.api_core.exceptions.InternalServerError: 500 Internal error encountered.

Note: I have tried with different network connections (to isolate firewall/proxy-related issues), but it is still getting reproduced each time.

[dpcollins-google]

Hello,

Apologies that you're having this issue. My guess is this is some other error being improperly masked to INTERNAL by our backend. Can you send an email to pubsub-lite-help@google.com (our support queue) with your actual project number so I can investigate further?

-Daniel

[dpcollins-google]

Already triaged. Moving to P3 and closing, lack of user response means it cannot be diagnosed.