DatastoreAdminClient.create_index does not successfully resolve long-running index operations.

[jlara310]

Thanks for stopping by to let us know something could be better!

If you are still having issues, please be sure to include as much information as possible:

Environment details

• OS type and version: Cloud Shell
• Python version: Python 3.9.2
• pip version: pip 20.3.4
• google-cloud-datastore version: 2.8.0

Steps to reproduce

Attempt to create an index using DatastoreAdminClient.create_index. See the example below.

Code example

import os
import time
from google.cloud import datastore_admin_v1

# Get default project
PROJECT = os.getenv('GOOGLE_CLOUD_PROJECT')
# Create an admin client
admin_client = datastore_admin_v1.DatastoreAdminClient()
# Create index for tag propery.
user_id_property = datastore_admin_v1.Index.IndexedProperty(
    name='userid',
    direction=datastore_admin_v1.Index.Direction.ASCENDING
    )
created_property = datastore_admin_v1.Index.IndexedProperty(
    name='created',
    direction=datastore_admin_v1.Index.Direction.ASCENDING
    )
index = datastore_admin_v1.Index(
    kind='Task',
    ancestor=datastore_admin_v1.Index.AncestorMode.NONE,
    properties=[user_id_property, created_property]
    )
request = datastore_admin_v1.CreateIndexRequest(project_id=PROJECT, index=index)
operation = admin_client.create_index(request=request)
print("Waiting for operation to complete...")
print(operation.metadata)
response = operation.result()
    
print(response)

Expected

I expect the index to be created and for the python script to synchronously wait for the result as described in https://googleapis.dev/python/google-api-core/latest/operation.html.

Observed

The index creation does go through and the index can be observed in the cloud console. However, the script errors out. Talked to a colleague who mentioned that the Datastore API might be eventually consistent here and the admin_client does not handle that.

Adding a wait to give time for the index to become visible to the client did get the script to function as expected. Is there any way the client can handle this on behalf of the user?

print("Waiting for operation to complete...")
print(operation.metadata)
while True:
    print(operation.metadata)
    time.sleep(.1)
    try:
        response = operation.result()
    except Exception as e:
        print(e)
    else:
        break

Stack trace

Waiting for operation to complete...
common {
  start_time {
    seconds: 1660091395
    nanos: 696000000
  }
  operation_type: CREATE_INDEX
  state: INITIALIZING
}
index_id: "CICAgNjp84oK"

Traceback (most recent call last):
  File "/home/juanlara/datastore/python/create-index/env/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 50, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/home/juanlara/datastore/python/create-index/env/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/juanlara/datastore/python/create-index/env/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.NOT_FOUND
        details = "Operation does not exist"
        debug_error_string = "{"created":"@1660091396.225852059","description":"Error received from peer ipv4:142.250.107.95:443","file":"src/core/lib/surface/call.cc","file_line":966,"grpc_message":"Operation does not exist","grpc_status":5}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/juanlara/datastore/python/create-index/create_index.py", line 27, in <module>
    response = operation.result()
  File "/home/juanlara/datastore/python/create-index/env/lib/python3.9/site-packages/google/api_core/future/polling.py", line 132, in result
    self._blocking_poll(timeout=timeout, **kwargs)
  File "/home/juanlara/datastore/python/create-index/env/lib/python3.9/site-packages/google/api_core/future/polling.py", line 110, in _blocking_poll
    retry_(self._done_or_raise)(**kwargs)
  File "/home/juanlara/datastore/python/create-index/env/lib/python3.9/site-packages/google/api_core/retry.py", line 283, in retry_wrapped_func
    return retry_target(
  File "/home/juanlara/datastore/python/create-index/env/lib/python3.9/site-packages/google/api_core/retry.py", line 190, in retry_target
    return target()
  File "/home/juanlara/datastore/python/create-index/env/lib/python3.9/site-packages/google/api_core/future/polling.py", line 88, in _done_or_raise
    if not self.done(**kwargs):
  File "/home/juanlara/datastore/python/create-index/env/lib/python3.9/site-packages/google/api_core/operation.py", line 170, in done
    self._refresh_and_update(retry)
  File "/home/juanlara/datastore/python/create-index/env/lib/python3.9/site-packages/google/api_core/operation.py", line 158, in _refresh_and_update
    self._operation = self._refresh(retry=retry)
  File "/home/juanlara/datastore/python/create-index/env/lib/python3.9/site-packages/google/api_core/operations_v1/operations_client.py", line 142, in get_operation
    return self._get_operation(
  File "/home/juanlara/datastore/python/create-index/env/lib/python3.9/site-packages/google/api_core/gapic_v1/method.py", line 154, in __call__
    return wrapped_func(*args, **kwargs)
  File "/home/juanlara/datastore/python/create-index/env/lib/python3.9/site-packages/google/api_core/retry.py", line 283, in retry_wrapped_func
    return retry_target(
  File "/home/juanlara/datastore/python/create-index/env/lib/python3.9/site-packages/google/api_core/retry.py", line 190, in retry_target
    return target()
  File "/home/juanlara/datastore/python/create-index/env/lib/python3.9/site-packages/google/api_core/timeout.py", line 210, in func_with_timeout
    return func(*args, **kwargs)
  File "/home/juanlara/datastore/python/create-index/env/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 52, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.NotFound: 404 Operation does not exist