Anyone facing Error 429? Client Error: Too Many Requests

Question

Anyone facing Error 429? Client Error: Too Many Requests

Closed this issue 2 months ago · 9 comments

I am receiving 429 error after 5 or 6 searches. I know the code does randomly choose a different user agent which shouldn't bring this error, however I did get it. I'm running my script from the command promt and wanted to check if other uses are also facing the same issue. I also do not see any Retry After in the error message which doesn't help.

I would request the community to provide any insights on this or workarounds if any.

Thanks in advance.

Answer 1 · 2023-09-10T13:56:48.000Z

I have the same problem of requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url. You may use time.sleep(0.06) in your code to slow down your query

Answer 2 · 2023-09-12T07:04:09.000Z

I have the same problem of requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url. You may use time.sleep(0.06) in your code to slow down your query

Thanks @jason-huanghao .

Answer 3 · 2023-09-30T02:27:41.000Z

I find out there are some queries that make the search function stall for over 1 minute, then they return 429, regardless of waiting time.
Ex. "Malaysia sugar tax, RM0.40 (US$0.086) per litre, more than 5 grams/100ml" takes a few seconds to retrieve the first 2 links, but at the 3rd, it makes me wait 1:30 mins, then returns 429, and the IP is unusable.
I tried the same query on Google Colab (that should not use my IP), yet, to be sure, I also tried switching internet connection to the phone hotspot and using an EC2, and all lead to the same results (breaking at the 3rd link of the same query): some queries can break the algorithm

Ideally, we should use the timeout params for the requests, but (I tried) it does not work in the case above. Any help is appreciated.

Answer 4 · 2023-09-30T07:08:04.000Z

One suggestion would be to randomize the user agent which is already done in the library but add more user agents and check if same user agent is not used previously. Adding a delay forcefully before a request can be another option. with regards Shuvro sent from my mobile device

…

On Sat, 30 Sep 2023, 4:27 am Michelangiolo Mazzeschi, < ***@***.***> wrote: I find out there are some queries that make the search function stall for over 1 minute, then they return 429, making the current IP unusuable. Ex. "Malaysia sugar tax, RM0.40 (US$0.086) per litre, more than 5 grams/100ml" takes a few seconds to retrieve the first 2 links, but at the 3rd, it makes me wait 1:30 mins, then returns 429, and the IP is unusable. I tried the same query on Google Colab (that should not use my IP), yet, to be sure, I also tried switching internet connection to the phone hotspot and using an EC2, and all lead to the same results (breaking at the 3rd link of the same query): some queries can break the algorithm Ideally, we would have a timeout for the requests, but I have not been able to implement it on the current llibrary. Any help is appreciated. — Reply to this email directly, view it on GitHub <#61 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATNSRLFGEGB55CFAUMN7C23X457STANCNFSM6AAAAAA36TX72E> . You are receiving this because you authored the thread.Message ID: ***@***.***>

Answer 5 · 2023-10-18T03:14:28.000Z

I am also experiencing the same error. I already tried forcefully adding a delay before a request but it is still returning the same 429 Client Error.

Can anybody help?

Answer 6 · 2023-12-07T15:56:16.000Z

Same question. How to fix the problem of too many requests?

Answer 7 · 2024-05-17T09:25:12.000Z

Google flags your code as a potential bot and the link actually shows a captcha to prove you're not. The best way to solve this is to just use their official Custom Search API as you're not allowed to use their search in the way this library does.

Answer 8 · 2024-05-24T18:35:59.000Z

The user agents are randomized but I believe there is a bug where it sends many queries for some searches. I will investigate this.

Answer 9 · 2024-08-03T23:06:24.000Z

I am not having this problem when I search "Malaysia sugar tax, RM0.40 (US$0.086) per litre, more than 5 grams/100ml", it sends 2 requests and finds this issue!