RetryError

[umax]

Hi, team!

Every day I get a lot of RetryErrors.

Environment details

• Official docker image python3.9-slim
• Python version: 3.9
• pip version: 21.0.1
• google-cloud-datastore version: 2.1.0

Code example

def get_account(restaurant_id):
    query = self._client.query(
        kind='DeliveryAccount',
        filters=(
            ('restaurant_id', '=', restaurant_id),
        ),
    )

    try:
        return list(query.fetch(limit=1))[0]
    except IndexError:
        return None

# self._client is a datastore.Client() instance cached for 45 minutes.

Stack trace

Deadline of 120.0s exceeded while calling functools.partial(<function _wrap_unary_errors.<locals>.error_remapped_callable at 0x3dff93540940>...

503 Getting metadata from plugin failed with error: HTTPSConnectionPool(host='oauth2.googleapis.com', port=443): Read timed out. (read timeout=120)

_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "Getting metadata from plugin failed with error: HTTPSConnectionPool(host='oauth2.googleapis.com', port=443): Read timed out. (read timeout=120)"
	debug_error_string = "{"created":"@1618478702.051101782","description":"Getting metadata from plugin failed with error: HTTPSConnectionPool(host='oauth2.googleapis.com', port=443): Read timed out. (read timeout=120)","file":"src/core/lib/security/credentials/plugin/plug...

Do you have any ideas why it happens or any suggestions how to fix these errors?

Thanks!

[craiglabenz]

Thanks for letting us know about this, @umax. I'm interested in some additional details about your situation to help us focus our search.

1. First, are you still seeing this error? The UNAVAILABLE error possibly suggests a transient server problem - though you did mention it happening "every day", so maybe not.
2. Do the errors always come from the exact code you shared, or is that merely a generic example of server errors that appear across your app?
3. How often does the erroring code run? (e.g., most page views from every user, once on client launch, etc?) If this is user accounts, I'm guessing it is closer to the latter.
4. Are you invalidating your self._client instance every 45 minutes? If so, what informed that decision?

[tseaver]

@umax Please feel free to reopen / follow up if you can provide the requested information to help is investigate this issue.