Add bigtable-attempt to request metadata when the RPC is a retry attempt
igorbernstein2 opened this issue · 2 comments
Thanks for stopping by to let us know something could be better!
Is your feature request related to a problem? Please describe.
Improve observability of network issues from the bigtable server side.
Describe the solution you'd like
When the initial RPC fails, the retry attempts that occur afterwards should populate bigtable-attempt
request metadata. The first retry attempt should set the value to 1. If that fails, the 2nd attempt should set it to 2, etc. Please note that the initial RPC should not set the header.
This will increase feature parity with java-bigtable.
Describe alternatives you've considered
Adding clientside metrics - would give us similar info, but will take a lot longer to implement and will have gaps due to possible connectivity issues with stackdriver
Logging - logging failed attempts can alert customers to retries occuring. However during support cases the customer would have to be aware of this
Thank you Igor! I think we can add support for this without too much trouble by modifying the readAll
in the ResumableStream
class
google-cloud-php/Bigtable/src/ResumableStream.php
Lines 139 to 142 in 7f1ceaf
Something like this should work:
// Send in "bigtable-attempt" header on each request
$headers = $this->callOptions['headers'] ?? [];
$headers['bigtable-attempt'] = ++$attempt;
// make RPC call
$stream = call_user_func_array(
[$this->gapicClient, $this->method],
[$this->request, ['headers' => $headers] + $this->callOptions]
);
Could you provide a link to the PR or documentation for the Java implementation so that we can compare and make sure they're consistent?
EDIT: I found the Java implementation here: googleapis/java-bigtable#935