Handling transport errors for transient network failures

[i386]

@marickvantuil I run into these class of transport layer errors on Google Cloud Run fairly often.

GuzzleHttp\Exception\ConnectException: cURL error 35: OpenSSL SSL_connect: Connection reset by peer in connection to cloudtasks.googleapis.com:443 (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://cloudtasks.googleapis.com/v2/projects/fabled-access-339904/locations/us-west1/queues/production-social-update-media-instagram?%24alt=json%3Benum-encoding%3Dint:

at .GuzzleHttp\Handler\CurlFactory::finishError ( [/var/www/html/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php:210](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fvar%2Fwww%2Fhtml%2Fvendor%2Fguzzlehttp%2Fguzzle%2Fsrc%2FHandler%2FCurlFactory.php&line=210&project=fabled-access-339904) )
at .GuzzleHttp\Handler\CurlFactory::finish ( [/var/www/html/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php:110](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fvar%2Fwww%2Fhtml%2Fvendor%2Fguzzlehttp%2Fguzzle%2Fsrc%2FHandler%2FCurlFactory.php&line=110&project=fabled-access-339904) )
at .GuzzleHttp\Handler\CurlMultiHandler->processMessages ( [/var/www/html/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:245](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fvar%2Fwww%2Fhtml%2Fvendor%2Fguzzlehttp%2Fguzzle%2Fsrc%2FHandler%2FCurlMultiHandler.php&line=245&project=fabled-access-339904) )
at .GuzzleHttp\Handler\CurlMultiHandler->tick ( [/var/www/html/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:169](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fvar%2Fwww%2Fhtml%2Fvendor%2Fguzzlehttp%2Fguzzle%2Fsrc%2FHandler%2FCurlMultiHandler.php&line=169&project=fabled-access-339904) )
at .GuzzleHttp\Handler\CurlMultiHandler->execute ( [/var/www/html/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:184](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fvar%2Fwww%2Fhtml%2Fvendor%2Fguzzlehttp%2Fguzzle%2Fsrc%2FHandler%2FCurlMultiHandler.php&line=184&project=fabled-access-339904) )
at .GuzzleHttp\Promise\Promise->invokeWaitFn ( [/var/www/html/vendor/guzzlehttp/promises/src/Promise.php:248](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fvar%2Fwww%2Fhtml%2Fvendor%2Fguzzlehttp%2Fpromises%2Fsrc%2FPromise.php&line=248&project=fabled-access-339904) )
at .GuzzleHttp\Promise\Promise->waitIfPending ( [/var/www/html/vendor/guzzlehttp/promises/src/Promise.php:224](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fvar%2Fwww%2Fhtml%2Fvendor%2Fguzzlehttp%2Fpromises%2Fsrc%2FPromise.php&line=224&project=fabled-access-339904) )
at .GuzzleHttp\Promise\Promise->invokeWaitList ( [/var/www/html/vendor/guzzlehttp/promises/src/Promise.php:269](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fvar%2Fwww%2Fhtml%2Fvendor%2Fguzzlehttp%2Fpromises%2Fsrc%2FPromise.php&line=269&project=fabled-access-339904) )
at .GuzzleHttp\Promise\Promise->waitIfPending ( [/var/www/html/vendor/guzzlehttp/promises/src/Promise.php:226](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fvar%2Fwww%2Fhtml%2Fvendor%2Fguzzlehttp%2Fpromises%2Fsrc%2FPromise.php&line=226&project=fabled-access-339904) )
at .GuzzleHttp\Promise\Promise->wait ( [/var/www/html/vendor/guzzlehttp/promises/src/Promise.php:62](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fvar%2Fwww%2Fhtml%2Fvendor%2Fguzzlehttp%2Fpromises%2Fsrc%2FPromise.php&line=62&project=fabled-access-339904) )
at .Google\Cloud\Tasks\V2\Gapic\CloudTasksGapicClient->getQueue ( [/var/www/html/vendor/google/cloud-tasks/src/V2/Gapic/CloudTasksGapicClient.php:703](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fvar%2Fwww%2Fhtml%2Fvendor%2Fgoogle%2Fcloud-tasks%2Fsrc%2FV2%2FGapic%2FCloudTasksGapicClient.php&line=703&project=fabled-access-339904) )
at .Stackkit\LaravelGoogleCloudTasksQueue\CloudTasksApiConcrete->getRetryConfig ( [/var/www/html/vendor/stackkit/laravel-google-cloud-tasks-queue/src/CloudTasksApiConcrete.php:29](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fvar%2Fwww%2Fhtml%2Fvendor%2Fstackkit%2Flaravel-google-cloud-tasks-queue%2Fsrc%2FCloudTasksApiConcrete.php&line=29&project=fabled-access-339904) )
at .Illuminate\Support\Facades\Facade::__callStatic ( [/var/www/html/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php:353](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fvar%2Fwww%2Fhtml%2Fvendor%2Flaravel%2Fframework%2Fsrc%2FIlluminate%2FSupport%2FFacades%2FFacade.php&line=353&project=fabled-access-339904) )
at .Stackkit\LaravelGoogleCloudTasksQueue\TaskHandler->loadQueueRetryConfig ( [/var/www/html/vendor/stackkit/laravel-google-cloud-tasks-queue/src/TaskHandler.php:174](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fvar%2Fwww%2Fhtml%2Fvendor%2Fstackkit%2Flaravel-google-cloud-tasks-queue%2Fsrc%2FTaskHandler.php&line=174&project=fabled-access-339904) )
at .Stackkit\LaravelGoogleCloudTasksQueue\TaskHandler->handleTask ( [/var/www/html/vendor/stackkit/laravel-google-cloud-tasks-queue/src/TaskHandler.php:124](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fvar%2Fwww%2Fhtml%2Fvendor%2Fstackkit%2Flaravel-google-cloud-tasks-queue%2Fsrc%2FTaskHandler.php&line=124&project=fabled-access-339904) )
at .Stackkit\LaravelGoogleCloudTasksQueue\TaskHandler->handle ( [/var/www/html/vendor/stackkit/laravel-google-cloud-tasks-queue/src/TaskHandler.php:57](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fvar%2Fwww%2Fhtml%2Fvendor%2Fstackkit%2Flaravel-google-cloud-tasks-queue%2Fsrc%2FTaskHandler.php&line=57&project=fabled-access-339904) )

For all of my guzzle usage, a retry with exponential backoff is configured like below and seems to help with recovery but I cant work out how to pass this to this driver. Any tips?

$handlerStack = HandlerStack::create(new CurlHandler());
        if ($qps) {
            $handlerStack->push(RateLimiterMiddleware::perSecond($qps));
        }
        $decider = function(int $retries, RequestInterface $request, ResponseInterface $response = null, \Throwable $reason = null) : bool {
            if ($retries >= MAX_RETRIES) {
                return false;
            }
            if ($response == null) {
                return true;
            }

            if ($reason instanceof TransferException) {
                return true;
            }

            return match ($response->getStatusCode()) {
                429, 500, 502, 503, 504 => true,
                default => false,
            };
        };
        $delay = fn() => RetryMiddleware::exponentialDelay(MAX_RETRIES);
        $handlerStack->push(Middleware::retry($decider, $delay));
        return new Client([
            'handler' => $handlerStack,
            'timeout' => $timeout,
            'headers' => ['Connection' => 'close'],
            CURLOPT_FORBID_REUSE => true,
            CURLOPT_FRESH_CONNECT => true,
        ]);

[marickvantuil]

In the CloudTasksServiceProvider I already add a singleton for the cloud tasks client:

$this->app->singleton(CloudTasksClient::class, function () {
    return new CloudTasksClient();
});

Maybe you can override it with a custom http handler?

$this->app->singleton(\Google\Cloud\Tasks\V2\CloudTasksClient::class, function () {
    return new \Google\Cloud\Tasks\V2\CloudTasksClient([
        'transportConfig' => [
            'rest' => [
                'httpHandler' => \Google\Auth\HttpHandler\HttpHandlerFactory::build(/* accepts A Guzzle\ClientInterface instance */)
            ],
        ]
    ]);
});

Not sure if this will work, or if the package needs a modification to make this possible.