google.api_core.exceptions.RetryError: Deadline of 60.0s exceeded while calling target function, last exception: 503 Getting metadata from plugin failed with error: ('invalid_grant: Bad Request', {'error': 'invalid_grant', 'error_description': 'Bad Request'})

Question

google.api_core.exceptions.RetryError: Deadline of 60.0s exceeded while calling target function, last exception: 503 Getting metadata from plugin failed with error: ('invalid_grant: Bad Request', {'error': 'invalid_grant', 'error_description': 'Bad Request'})

kyegomez opened this issue 2 years ago · 23 comments

Description of the bug:

package verison: 0.3.1

  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/api_core/grpc_helpers.py", line 65, in error_remapped_callable
    return callable_(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/grpc/_channel.py", line 1161, in __call__
    return _end_unary_response_blocking(state, call, False, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/grpc/_channel.py", line 1004, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "Getting metadata from plugin failed with error: ('invalid_grant: Bad Request', {'error': 'invalid_grant', 'error_description': 'Bad Request'})"
        debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2023-12-13T23:10:10.374012-08:00", grpc_status:14, grpc_message:"Getting metadata from plugin failed with error: (\'invalid_grant: Bad Request\', {\'error\': \'invalid_grant\', \'error_description\': \'Bad Request\'})"}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/api_core/retry.py", line 191, in retry_target
    return target()
           ^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.ServiceUnavailable: 503 Getting metadata from plugin failed with error: ('invalid_grant: Bad Request', {'error': 'invalid_grant', 'error_description': 'Bad Request'})

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/defalt/Desktop/Athena/research/swarms/gemini.py", line 23, in <module>
    response = model.generate_content("The opposite of hot is")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/generativeai/generative_models.py", line 243, in generate_content
    response = self._client.generate_content(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/ai/generativelanguage_v1beta/services/generative_service/client.py", line 566, in generate_content
    response = rpc(
               ^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/api_core/gapic_v1/method.py", line 113, in __call__
    return wrapped_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/api_core/retry.py", line 349, in retry_wrapped_func
    return retry_target(
           ^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/api_core/retry.py", line 207, in retry_target
    raise exceptions.RetryError(
google.api_core.exceptions.RetryError: Deadline of 60.0s exceeded while calling target function, last exception: 503 Getting metadata from plugin failed with error: ('invalid_grant: Bad Request', {'error': 'invalid_grant', 'error_description': 'Bad Request'})

Actual vs expected behavior:

No response

Any other information you'd like to share?

No response

Answer 1 · 2023-12-14T08:40:12.000Z

I also encountered this problem GRPC returns 503, but my error is google.api_core.exceptions.RetryError: Deadline of 60.0s exceeded while calling target function, last exception: 503 failed to connect to all addresses; last error: UNKNOWN: ipv4 :172.217.160.74:443: Failed to connect to remote host: FD Shutdown. IP cannot be accessed

Answer 2 · 2023-12-14T08:55:07.000Z

huqin-RM commented 2 years ago

Answer 3 · 2023-12-16T14:21:53.000Z

The same problem encountered by me

Answer 4 · 2023-12-18T01:33:40.000Z

You can bypass GRPC and use rest:

genai.configure(api_key=GOOGLE_API_KEY,transport='rest')

Answer 5 · 2023-12-18T05:51:46.000Z

@huqin-RM thanks that worked, I'm integrating it into the Swarms framework that enables developers to Build, Deploy, and Scale Reliable Autonomous Agents!

https://www.swarms.world/

And, here is the docs for our abstracted away Gemini wrapper:
https://swarms.apac.ai/en/latest/swarms/models/gemini/

Answer 6 · 2023-12-18T08:42:26.000Z

This works, thank you very much

Answer 7 · 2023-12-18T08:44:05.000Z

It really works.

Answer 8 · 2023-12-18T09:16:18.000Z

Thank you, it's worked. And I still want to know why you know this way can work?

Answer 9 · 2023-12-18T10:46:14.000Z

Answer 10 · 2023-12-19T05:54:12.000Z

Thank you. You have a clear way to solve problem.

Answer 11 · 2023-12-24T04:29:18.000Z

Thanks a lot.It works and solves a problem that has been bothering me for three weeks.

Answer 12 · 2024-01-18T14:29:59.000Z

I/m having this error msg now:
Traceback (most recent call last):
File "/Users/hulksi/Desktop/upload genreList/update(genreList).py", line 13, in
genai.configure(api_key=GOOGLE_API_KEY, transport='rest')
AttributeError: module 'genai' has no attribute 'configure'

Answer 13 · 2024-02-08T10:43:30.000Z

In my case the problem was docker, I reinstalled the docker to fix the problem (though I acknowledge there might be a better solution)

Answer 14 · 2024-03-01T07:54:39.000Z

The problem with GRPC is still same even after two months.
But this transport='rest' does solve the problem. Thanks

Hoping that the problem with GRPC will be solved soon. Facing same issue with TextGeneration on VertexAI as well.

Answer 15 · 2024-03-15T05:21:06.000Z

This solved my issue😄. You should replace port with your proxy server's port number:

import os

os.environ['http_proxy'] = 'http://127.0.0.1:port'
os.environ['https_proxy'] = 'http://127.0.0.1:port'
os.environ['all_proxy'] = 'socks5://127.0.0.1:port'

Answer 16 · 2024-03-26T09:20:07.000Z

Thanks for sharing! The problem that has bothered me for a long time does not seem to be found in the official document 😂

Answer 17 · 2024-04-07T10:12:03.000Z

genai.configure（api_key=GOOGLE_API_KEY，transport='rest'）
The problem can be solved, but the streaming Q&A output will fail.

Answer 18 · 2024-06-01T13:18:24.000Z

Thanks! It really works.

Answer 19 · 2024-08-27T23:16:40.000Z

Lots happened in this thread - is there anything left we can help with? If not, we'll close this one.

Answer 20 · 2024-09-11T01:56:39.000Z

Marking this issue as stale since it has been open for 14 days with no activity. This issue will be closed if no further activity occurs.

Answer 21 · 2024-09-25T02:00:35.000Z

This issue was closed because it has been inactive for 28 days. Please post a new issue if you need further assistance. Thanks!

Answer 22 · 2025-02-10T07:23:08.000Z

It works, thank you.