PMassicotte/gtrendsR

Status code 429

juansqw opened this issue · 30 comments

juansqw commented

Im trying to run an old script where I get the gtrends of 10 terms with a sleep time of 3-5 seconds. However, I can barely make 4 queries before I get the 429 status code.

Was there an update in the search limits? If so, how are they now?

Here's a repex of what im trying to do:

terms <- c("calderas + compra", "spa", "Cosmetology & Beauty Professionals", 
           "equipos + electricos", "ars humano", "Insurance", "caribe group + santo domingo", 
           "precios + cobre", "importadora + madera", "Electronics & Electrical"
)

searches <- map(terms, gtrends)
PMassicotte commented

The first thing I notice is that some keywords return nothing:

image

PMassicotte commented

You also my want to deal with queries that do not return something:

> for (i in terms) {
+   print(i)
+   
+   try(gtrends(i))
+   Sys.sleep(1)
+ }
[1] "calderas + compra"
[1] "spa"
[1] "Cosmetology & Beauty Professionals"
Error : No data returned by the query. Consider changing search parameters.
[1] "equipos + electricos"
[1] "ars humano"
[1] "Insurance"
[1] "caribe group + santo domingo"
[1] "precios + cobre"
[1] "importadora + madera"
[1] "Electronics & Electrical"
bpffjl commented

I'm running into a similar issue with this code.

g <-
gtrends(keyword = "food stamps", geo = "US-MN", time = "today 12-m",
gprop = "web")$interest_over_time

I run this daily and it has worked mostly well over the past two years. Recently I've been getting this error:

Status code was not 200. Returned status code:429

I've tried the code again with more popular keywords like "NFL" and still get the error.

However after a few tries it just works with no changes to the code.

Any thoughts on why that might be happening?

PMassicotte commented

Interesting, it seems that 429 is related to too many requests:

https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429

BillPetti commented

I'm also seeing this. I've tried multiple queries of different terms from home network, from remotely behind a corporate VPN, and from a custom EC2 instance of R and experience the same issues.

cspenn commented

I've run into this a lot. My workaround has been to budget ample time and make multiple runs with a VPN, changing endpoints each run, and then collating the results into a master file. I'd love to figure out a way, with a service like NordVPN, where I can simply change endpoints per request the moment a request hits a 429, but I'm not sure they have an API that supports that.

  • 👍2
btoblerone commented

Did anyone figure out a solution to this? I'm having the same issue.

RiiNagaja commented

This is still an issue to this day. I used multiple IP addresses via VPN as well as my own, and get the 429 error code even with the simplest .build_payload request. That means the "too many requests" thing is an illusionary error message, since it will just fail by default on the first try.
The code I used worked fine for months until this week, and the only thing I did was install a newer python version (3.10.6 to be exact), as well as set up all modules freshly of course. Therefore it might be a fair guess that the pytrends module is simply incompatible with higher versions.

RajaBellebon commented

Hello, apparently, Google changed the portal and the API, can anyone confirm that the code as per today is working?
I try a super simple script like this

library(gtrendsR) 
trends <- gtrends("Biden", 
                  gprop = 'web',
                  geo = 'US',
                  time = "2023-02-15 2023-04-14" )

I receive

Error in FUN(X[[i]], ...) : 
  Status code was not 200. Returned status code:429
JBleher commented

for me the following works

gtrends("Biden", 
                  gprop = 'web',
                  geo = 'US',
                  time = "2023-02-15 2023-04-14", onlyInterest = TRUE )

the problem is that the function related_topics or related_queries fails. This is imo because they are called to fast in succession. There simply needs to be a Sys.wait() between the calls. Then everything else should work as the usual. API hasn't changed imo.

PMassicotte commented

Also works on my side.

  • 👍1
RajaBellebon commented

API hasn't changed imo.

https://searchengineland.com/google-launches-new-google-trends-portal-394026
I thought it did, let me try again and thanks for your comments

bachmannpatrick commented

I have the same issue her. gtrends() is no longer reliable. Sometimes everything works fine, but more often I get Status code was not 200. Returned status code:429. It used to work perfectly a month a go.

Sample code:
gtrends("NFL", geo="US", gprop = "web", time = "today+5-y", low_search_volume = T, onlyInterest = T)

l-eadz commented

I've come across the same issue with this. I used to get 429 errors using a similar process in Python, but it was reliable in R. I'm now getting 429 errors using gtrendsR

RajaBellebon commented

As I mentioned in my comment above, Google may have change some stuff in their API for the trends portal but it was not fully confirmed yet