Sporadic "Invalid Credentials" Issues - Service Account

Question

Sporadic "Invalid Credentials" Issues - Service Account

strategyst opened this issue 8 years ago · 22 comments

I seem to be having weird issues with "Invalid Credentials" errors, service accounts and the Search Console API. Sometimes everything works and I can access the API, then some time later I may try an access the SC API and I get an "Invalid Credentials" error. This is without changing any code.

Edit: Very weird. I swapped around my scopes and then it started working again. Nothing else was changed. Why would swapping my scopes around every so often work?

Here is what I am using:

class Google {

    private function googleApiAuthorise()
    {

        $client = new \Google_Client();
        putenv('GOOGLE_APPLICATION_CREDENTIALS=' . base_path() . '/keys/Tool-xxxxxxxxx.json');
        $client->useApplicationDefaultCredentials();
        $client->setSubject(example@example.iam.gserviceaccount.com');
        $client->setApplicationName("Tool");
        $scopes = ['https://www.googleapis.com/auth/webmasters.readonly', 'https://www.googleapis.com/auth/webmasters'];
        $client->setScopes($scopes);

        if( $client->isAccessTokenExpired()) {

            $client->refreshTokenWithAssertion();

        }

        return $client;

    }

    public function getSearchAnalytics($search_type = 'web')
    {

        $authorise = Google::googleApiAuthorise();

        $service = new \Google_Service_Webmasters($authorise);

        $request = new \Google_Service_Webmasters_SearchAnalyticsQueryRequest;

        $request->setStartRow(0);
        $request->setStartDate('2016-06-01');
        $request->setEndDate(Carbon::now()->toDateString());
        $request->setSearchType($search_type);
        $request->setRowLimit(200);
        $request->setDimensions(array('date','page','country','device','query'));

        $query_search = $service->searchanalytics->query("http://www.example.com/", $request); 
        $rows = $query_search->getRows();

        return $rows;

    }
}

The weird thing is it works sometimes. Is this some sort of bug? I've followed pretty much every tutorial out there about service accounts and this API client, but it still seems to be quite random as to why sometimes I can use the API and other times I get an "Invalid Credentials" error.

Answer 1 · 2016-10-15T15:26:01.000Z

When you say "sporadic" do you mean after 3600 seconds? That would be the time when the access token is expired and you would have to get a new one.

I am having the same issue and i do understand that it is due to the expired access token. This is my particular code sequence (which is quite similar to yours):

    /* This env variable holds the path to the keyfile with the JSON based certificate. It will be required by the Google API Client */
    putenv( 'GOOGLE_APPLICATION_CREDENTIALS=' . $objApplication->getConfig( 'App::GoogleAuthServiceAccoutKeyAsJson' ) );
    try{

        /* Using Stash for caching */
        //$objCache = new Pool( new FileSystem );

        //$objCacheLogger = new Logger;
        //$hdTokenCallback = function( $cacheKey, $accessToken ) use ( $objCacheLogger ) {
        //    $objCacheLogger->debug( sprintf( 'A new access token \'%s\' has beeen received for the cache key %s', $accessToken, $cacheKey ) );
        //};

        /* Creating the client */
        $objGoogleApiClient = new \Google_Client();
        $objGoogleApiClient->useApplicationDefaultCredentials();
        $objGoogleApiClient->setApplicationName( 'myTestApplication' );
        $objGoogleApiClient->setAccessType( 'offline' );
        $objGoogleApiClient->setScopes( [ 'https://www.googleapis.com/auth/webmasters.readonly' ] );
        //$objGoogleApiClient->setCache( $objCache );
        //$objGoogleApiClient->setTokenCallback( $hdTokenCallback );

        /* Refresh token when expired */
        if( $objGoogleApiClient->isAccessTokenExpired() ){
            $objGoogleApiClient->refreshTokenWithAssertion();
        }

        /* Creating the actual service */
        $this->objGoogleService = new \Google_Service_Webmasters( $objGoogleApiClient );

    } catch( \Google_Exception $objException ){
        die(" An exception has occured!" );
    }

I have also diggged into what feels like the entire documentation there is in the Internet ;-). But i still get the expiration after 3600. I am as well using a service account and this code runs on the console.

What am i doing wrong, why am getting the expiration? Is this part

        /* Refresh token when expired */
        if( $objGoogleApiClient->isAccessTokenExpired() ){
            $objGoogleApiClient->refreshTokenWithAssertion();
        }

not enough to do the refreshing? I used a debugger but could not find a refresh token passed back to Client.php, it was always null. Recreated millions of service accounts and went through the initial authentication process expecting to get one, but none which i could save away and use for later refreshing process.

As opposed to omnicodagithub i am getting the 401 not sporadically but regularly after one hour.

Above code is using google/apiclient v2.0.3.

Thanks for any advice.

Answer 2 · 2016-10-15T16:54:48.000Z

It was the expiration. I've decided to go with OAuth instead but that's also seeming to be a nightmare to set up. Even when I copy Google's own examples they often don't seem to work or only work temporarily.

Answer 3 · 2016-10-16T08:54:50.000Z

-- Even when I copy Google's own examples they often don't seem to work or only work temporarily
Which is why i want to solve this approach here now because i already spent too much time trying to resolve this. Version 1 of the API worked fine (at least for the Search Console) but i was missing the setStartRow() for result sets exceeding 5000 rows in the old version. Along with the composer support it makes sense for me to update to version 2.

Answer 4 · 2016-10-16T13:12:20.000Z

Update:
Just for the heck of it i downgraded the Google Api Client to v.2.0.2 and am realizing that above code works perfectly, also after 3600 seconds have been exceeded. Refreshing the access token works just fine. So without having gone into depth the mentioned misbehaviour only occurs with the Api Client in v.2.0.3.

Answer 5 · 2016-10-17T07:22:19.000Z

We have the same issue with v2.0.3

if ($client->isAccessTokenExpired()) {
    $client->fetchAccessTokenWithRefreshToken($refreshToken);
}

We see expired access token and ~30 seconds after fetching a new access token the API responds with "Invalid Credentials".

Re-initialising the client when the access token expires works as a current "dirty but worky" solution.

Answer 6 · 2016-10-17T17:42:47.000Z

@omnicodagithub I had the same issue, but I think I have a solution, @bshaffer does some checks for cache in the library prebuilt, can you try this?

  $cache = new Stash\Pool(new Stash\Driver\FileSystem); // or whatever
  $client->setCache($cache);

   if( $client->isAccessTokenExpired() ){
        $cache->clear();
         $client->refreshTokenWithAssertion();
   }

Answer 7 · 2016-10-17T22:52:53.000Z

I am looking into this. Can you provide a stack trace for the Invalid Credentials error?

I believe the problem may be the refresh token is not being persisted in the in-memory cache.

Answer 8 · 2016-10-17T23:56:15.000Z

Changing the scopes uses a different cache key. This is almost certainly why that fixes the issue. But if you're using in-memory cache, I would think the tokens wouldn't persist enough for it to matter.

Answer 9 · 2016-10-18T08:50:45.000Z

Links to another question on stack along the same lines. Google Analytics Reporting API V4 with “Request had invalid authentication credentials”

Answer 10 · 2016-10-18T20:56:14.000Z

Just an update from me. So to test I am storing the entire access token in the database (access_key, token_type, expires_in, refresh_key and created). I am using this token from the database with the setAccessToken method. This is almost always resulting in a 401 "Invalid Credentials" error. When I test this same key in the Google API Playground it works fine, so it doesn't seem to be the access token.

Not sure if I'm not quite understanding it, but if I'm using an access token from the database, then the cache shouldn't affect this should it? By the way, it did actually work earlier for a while using this method. This issue really has me scratching my head as one time it's working fine, the next it's not working.

Answer 11 · 2016-10-18T21:19:55.000Z

Thanks for the update @omnicodagithub ! Ahh how the plot thickens.

Could you do two things for me please?

Comment here with a stack trace.
Use Charles Proxy to obtain the HTTP request of the call, and ensure that the requests going to www.googleapis.com/auth (to retrieve access tokens) and the requests going to www.googleapis.com/API_NAME (to make the api call) match what you'd expect, i.e. the client credentials, access token from the database, etc.

Set up the proxy by adding this when you create your client:

$options = [
    'proxy' => 'localhost:8888', // the default for charles
    'verify' => false,
];
$httpClient = new GuzzleHttp\Client($options);
$client->setHttpClient($httpClient);

Answer 12 · 2016-10-18T23:38:07.000Z

Hi @bshaffer! Thanks for the help by the way.

Ok, so here is the stack trace:

Stack trace:
#0 C:\xampp\htdocs\marketingtool\vendor\google\apiclient\src\Google\Http\REST.php(94): Google_Http_REST::decodeHttpResponse(Object(GuzzleHttp\Psr7\Response), Object(GuzzleHttp\Psr7\Request), 'Google_Service_...')
#1 [internal function]: Google_Http_REST::doExecute(Object(GuzzleHttp\Client), Object(GuzzleHttp\Psr7\Request), 'Google_Service_...')
#2 C:\xampp\htdocs\marketingtool\vendor\google\apiclient\src\Google\Task\Runner.php(181): call_user_func_array(Array, Array)
#3 C:\xampp\htdocs\marketingtool\vendor\google\apiclient\src\Google\Http\REST.php(58): Google_Task_Runner->run()
#4 C:\xampp\htdocs\marketingtool\vendor\google\apiclient\src\Google\Client.php(781): Google_Http_REST::execute(Object(GuzzleHttp\Client), Object(GuzzleHttp\Psr7\Request), 'Google_Service_...', Array)
#5 C:\xampp\htdocs\marketingtool\vendor\google\apiclient\src\Google\Service\Resource.php(232): Google_Client->execute(Object(GuzzleHttp\Psr7\Request), 'Google_Service_...')
#6 C:\xampp\htdocs\marketingtool\vendor\google\apiclient-services\src\Google\Service\Webmasters\Resource\Searchanalytics.php(48): Google_Service_Resource->call('query', Array, 'Google_Service_...')
#7 C:\xampp\htdocs\marketingtool\app\Http\Controllers\SearchConsoleController.php(146): Google_Service_Webmasters_Resource_Searchanalytics->query('http://www.omni...', Object(Google_Service_Webmasters_SearchAnalyticsQueryRequest))
#8 [internal function]: App\Http\Controllers\SearchConsoleController->test()
#9 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Routing\Controller.php(55): call_user_func_array(Array, Array)
#10 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Routing\ControllerDispatcher.php(44): Illuminate\Routing\Controller->callAction('test', Array)
#11 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Routing\Route.php(190): Illuminate\Routing\ControllerDispatcher->dispatch(Object(Illuminate\Routing\Route), Object(App\Http\Controllers\SearchConsoleController), 'test')
#12 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Routing\Route.php(144): Illuminate\Routing\Route->runController()
#13 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Routing\Router.php(642): Illuminate\Routing\Route->run(Object(Illuminate\Http\Request))
#14 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Routing\Pipeline.php(53): Illuminate\Routing\Router->Illuminate\Routing{closure}(Object(Illuminate\Http\Request))
#15 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Routing\Middleware\SubstituteBindings.php(41): Illuminate\Routing\Pipeline->Illuminate\Routing{closure}(Object(Illuminate\Http\Request))
#16 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Pipeline\Pipeline.php(137): Illuminate\Routing\Middleware\SubstituteBindings->handle(Object(Illuminate\Http\Request), Object(Closure))
#17 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Routing\Pipeline.php(33): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}(Object(Illuminate\Http\Request))
#18 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Auth\Middleware\Authenticate.php(43): Illuminate\Routing\Pipeline->Illuminate\Routing{closure}(Object(Illuminate\Http\Request))
#19 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Pipeline\Pipeline.php(137): Illuminate\Auth\Middleware\Authenticate->handle(Object(Illuminate\Http\Request), Object(Closure))
#20 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Routing\Pipeline.php(33): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}(Object(Illuminate\Http\Request))
#21 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Foundation\Http\Middleware\VerifyCsrfToken.php(65): Illuminate\Routing\Pipeline->Illuminate\Routing{closure}(Object(Illuminate\Http\Request))
#22 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Pipeline\Pipeline.php(137): Illuminate\Foundation\Http\Middleware\VerifyCsrfToken->handle(Object(Illuminate\Http\Request), Object(Closure))
#23 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Routing\Pipeline.php(33): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}(Object(Illuminate\Http\Request))
#24 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\View\Middleware\ShareErrorsFromSession.php(49): Illuminate\Routing\Pipeline->Illuminate\Routing{closure}(Object(Illuminate\Http\Request))
#25 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Pipeline\Pipeline.php(137): Illuminate\View\Middleware\ShareErrorsFromSession->handle(Object(Illuminate\Http\Request), Object(Closure))
#26 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Routing\Pipeline.php(33): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}(Object(Illuminate\Http\Request))
#27 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Session\Middleware\StartSession.php(64): Illuminate\Routing\Pipeline->Illuminate\Routing{closure}(Object(Illuminate\Http\Request))
#28 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Pipeline\Pipeline.php(137): Illuminate\Session\Middleware\StartSession->handle(Object(Illuminate\Http\Request), Object(Closure))
#29 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Routing\Pipeline.php(33): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}(Object(Illuminate\Http\Request))
#30 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Cookie\Middleware\AddQueuedCookiesToResponse.php(37): Illuminate\Routing\Pipeline->Illuminate\Routing{closure}(Object(Illuminate\Http\Request))
#31 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Pipeline\Pipeline.php(137): Illuminate\Cookie\Middleware\AddQueuedCookiesToResponse->handle(Object(Illuminate\Http\Request), Object(Closure))
#32 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Routing\Pipeline.php(33): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}(Object(Illuminate\Http\Request))
#33 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Cookie\Middleware\EncryptCookies.php(59): Illuminate\Routing\Pipeline->Illuminate\Routing{closure}(Object(Illuminate\Http\Request))
#34 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Pipeline\Pipeline.php(137): Illuminate\Cookie\Middleware\EncryptCookies->handle(Object(Illuminate\Http\Request), Object(Closure))
#35 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Routing\Pipeline.php(33): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}(Object(Illuminate\Http\Request))
#36 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Pipeline\Pipeline.php(104): Illuminate\Routing\Pipeline->Illuminate\Routing{closure}(Object(Illuminate\Http\Request))
#37 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Routing\Router.php(644): Illuminate\Pipeline\Pipeline->then(Object(Closure))
#38 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Routing\Router.php(618): Illuminate\Routing\Router->runRouteWithinStack(Object(Illuminate\Routing\Route), Object(Illuminate\Http\Request))
#39 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Routing\Router.php(596): Illuminate\Routing\Router->dispatchToRoute(Object(Illuminate\Http\Request))
#40 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Foundation\Http\Kernel.php(267): Illuminate\Routing\Router->dispatch(Object(Illuminate\Http\Request))
#41 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Routing\Pipeline.php(53): Illuminate\Foundation\Http\Kernel->Illuminate\Foundation\Http{closure}(Object(Illuminate\Http\Request))
#42 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Foundation\Http\Middleware\CheckForMaintenanceMode.php(46): Illuminate\Routing\Pipeline->Illuminate\Routing{closure}(Object(Illuminate\Http\Request))
#43 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Pipeline\Pipeline.php(137): Illuminate\Foundation\Http\Middleware\CheckForMaintenanceMode->handle(Object(Illuminate\Http\Request), Object(Closure))
#44 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Routing\Pipeline.php(33): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}(Object(Illuminate\Http\Request))
#45 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Pipeline\Pipeline.php(104): Illuminate\Routing\Pipeline->Illuminate\Routing{closure}(Object(Illuminate\Http\Request))
#46 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Foundation\Http\Kernel.php(149): Illuminate\Pipeline\Pipeline->then(Object(Closure))
#47 C:\xampp\htdocs\marketingtool\vendor\laravel\framework\src\Illuminate\Foundation\Http\Kernel.php(116): Illuminate\Foundation\Http\Kernel->sendRequestThroughRouter(Object(Illuminate\Http\Request))
#48 C:\xampp\htdocs\marketingtool\public\index.php(54): Illuminate\Foundation\Http\Kernel->handle(Object(Illuminate\Http\Request))
#49 {main}

Regarding point two, when using Charles I can see that a different access_token is being used for authorisation. The one I am using for setAccessToken() is definitely the correct one (I'm printing it out to the page so I can confirm). Is there a cached one and is that one being used instead of the one I'm passing to setAccessToken()?

Answer 13 · 2016-10-19T00:39:15.000Z

@omnicodagithub hmm, if you've told the Google_Client object to use application default credentials, it will use those instead of the access token. In your code above you call useApplicationDefaultCredentials. Are you calling $client->setAccessToken in the same call?

Digging deeper into this, it appears that the explicitly-set Access Token bypasses the cache, and so shouldn't be affected.

Is it possible you're using two different service accounts? Have you tried rolling back to v2.0.2 as suggested above?

Answer 14 · 2016-10-19T02:52:25.000Z

@bshaffer looking at the github compare with a previous stable version; It seems cache is set always, in memory or filesystem if pool library is available

This is why clearing works, since when you clear it makes a brand new token refresh call (which defeats the whole purpose of cache)

Hope this was helpful, maybe I have some time tomorrow I can submit a pull request for the line to check if whats in cache is expired first before using it

Answer 15 · 2016-10-19T09:35:03.000Z

@bshaffer Hmm weird. I was using useApplicationDefaultCredentialsbefore, but I stopped using it days ago. I was originally using service accounts, but I decided instead to go with just OAauth. I'm using setAuthConfig, but that only looks as though it uses useApplicationDefaultCredentials if the JSON file has a type of 'service_account'.

I added this last night, just to test, so it could be this as the reason for useApplicationDefaultCredentials:

if( $client->isAccessTokenExpired() ){
        $cache->clear();
         $client->refreshTokenWithAssertion();
   }

Edit: I did what @chonthu suggested and I just cleared the cache. I started getting results again after that.

Answer 16 · 2016-10-19T19:42:45.000Z

You shouldn't need to manually clear the cache. This implies the cache is not invalidated properly. Looking in the code, a token being received should expire after 1500 seconds.

Can you call var_dump(get_class($client->getCache())); and tell me what you get?

Answer 17 · 2016-10-20T21:18:55.000Z

To fix this in the auth library, I've added googleapis/google-auth-library-php#138, which is not ideal (the call should be unnecessary) but will get the job done.

Another option would be to stop using tedivm/stash by default, which is what we should have done to start, but it would break B.C.

Answer 18 · 2017-03-17T15:08:35.000Z

I just wanted to mention that we're still seeing this error in 2.0.3 (as well as the last 1.x release). The only thing that worked was manually fetching a new access token, as mentioned above. My hack is to do that on the first 401 and retry the API call but I would obviously prefer if this just worked.

Please let me know if there's anything you'd like me to provide. I can send stack traces, and I'm using the Calendar API.

Answer 19 · 2017-05-23T20:42:04.000Z

I ran into this problem and have been working around it for months; finally got around to tracing down this bug. The Stash library's response appears to include expired cache data which is properly flagged as a Miss (which could be a benefit for some situations), however the Google Auth library never tests it to confirm it should be used. This pull request does a proper verification of the response; and allows us to replace hacks like the one suggested in October by @chonthu (which was clearing the entire Stash cache on every hit). My client setup code now no longer needs to test for expiration or manually refresh the tokens.

Answer 20 · 2017-06-12T23:01:27.000Z

@BVMiko while i agree with this fix because of the quote from PSR-6, it looks like stash ensures the isHit property is false before returning anything with get, so that either way the function returns null (see here). So at least with the Stash implementation, I do not see how this will fix the issues described.

Answer 21 · 2017-06-12T23:02:55.000Z

Looks like I added that 8 months ago...

Answer 22 · 2017-06-12T23:09:39.000Z

Okay, I just figured out what's going on. Two weeks before your PR, a new version of Stash was finally tagged with the fix I added 8 months ago. So this is now fixed with v0.14.2.

Since the PSR-6 spec says the calling library SHOULD verify the cache hit, the changes in #150 are valid, even though the spec ALSO says get should return NULL if there is no hit (making the call to isHit unnecessary).

Either way this issue is fixed!