geduldig/TwitterAPI

APIv2 tweets pager - TwitterRequestError

factorlive opened this issue · 15 comments

factorlive commented

Thanks again for implementing an enhancement based on an issue ( #167).

Unfortunately, there is a knock-on effect on the following API v2 endpoint:

tweets/search/recent

Code

api = TwitterAPI(os.getenv("CONSUMER_KEY"), os.getenv("CONSUMER_SECRET"),
                 os.getenv("TOKEN"), os.getenv("SECRET"), api_version='2')
pager = TwitterPager(api, 'tweets/search/recent', {'query':"Twitterdev"})
for item in pager.get_iterator(new_tweets=False):
    print(item)

Output and error:

{'id': '1362416352351244293', 'text': 'RT @TwitterDev: Learn how academics can get historical Tweets using the full-archive search endpoint v2 available in the Academic Research…'}
{'id': '1362413374764630021', 'text': 'Transferowe wydarzenie ostatnich dni to podpisanie przez stopera ŁKS Jana Sobocińskiego  trzyipółletniego kontraktu z... amerykańską drużyną Charlotte FC, która pojawi się w MLS w 2022 roku. https://t.co/3iQqLtH0it #example # demo przez @twitterdev'}
{'id': '1362409301860622341', 'text': 'Who makes the rules? @TwitterDev? @TwitterComms? @Twitter policy folks? How much weight will disinformation researchers, reporters and fact-checkers get in deciding how to set the algorithm for the rated helpful channel?'}
{'id': '1362405652430286851', 'text': 'Z dr. med. Januszem Szelugą, specjalistą psychiatrą z 40-letnim doświadczeniem zawodowym, rozmawia Dorota Abramowicz https://t.co/SPDOh4YIPN #example # demo przez @twitterdev \nDR JANUSZ SZELUGA'}
{'id': '1362397547382251521', 'text': '@TwitterDev Siap laksanakan'}
{'id': '1362390722129846276', 'text': 'Hello world https://t.co/yWDZZgiM9Z #demo via @twitterdev'}
{'id': '1362378944218071045', 'text': 'W zderzeniu dwóch osobówek zginęły dwie osoby. Jedną\xa0z ofiar jest Dominik Grabas - właściciel pizzerii w Rzeszowie. W Internecie trwa zbiórka pieniędzy dla jego rodziny. https://t.co/Uf2UkhzyWX #example # demo przez @twitterdev'}
{'id': '1362377262692143104', 'text': "Twitter is just full of bloat honestly, I don't like topics, I don't like fleets, I don't like getting recommended stuff, and I don't like getting notified about someone's tweet all of a sudden. Sure ads are fine, but the rest is just absurd @jack @TwitterDesign @TwitterDev"}
{'id': '1362365168894676998', 'text': '@tinucherian @Twitter @manishm @TwitterDev @jack I see it on my profile. Probably something they  might be trying or enhancing on the verified handles ?'}
{'id': '1362364357624102912', 'text': 'Nice. If you ever fancy joining forces to defeat @DnD_Lich then let me know! https://t.co/tYOEIeRbf8'}
---------------------------------------------------------------------------
TwitterRequestError                       Traceback (most recent call last)
<ipython-input-21-9810d1ec73a1> in <module>
      2                  os.getenv("TOKEN"), os.getenv("SECRET"), api_version='2')
      3 pager = TwitterPager(api, 'tweets/search/recent', {'query':"Twitterdev"})
----> 4 for item in pager.get_iterator(new_tweets=False):
      5     print(item)

~/.cache/pypoetry/virtualenvs/campaign-IjD-JE0s-py3.9/lib/python3.9/site-packages/TwitterAPI/TwitterPager.py in get_iterator(self, wait, new_tweets)
     47                 start = time.time()
     48                 r = self.api.request(self.resource, self.params)
---> 49                 it = r.get_iterator()
     50                 if new_tweets:
     51                     it = reversed(list(it))

~/.cache/pypoetry/virtualenvs/campaign-IjD-JE0s-py3.9/lib/python3.9/site-packages/TwitterAPI/TwitterAPI.py in get_iterator(self)
    238         """
    239         if self.response.status_code != 200:
--> 240             raise TwitterRequestError(self.response.status_code, msg=self.response.text)
    241 
    242         if self.stream:

TwitterRequestError: ('{"errors":[{"parameters":{"pagination_token":["b26v89c19zqg8o3fosns2v373jl0fj3jxys6lrnxm8h31"]},"message":"The query parameter [pagination_token] is not one of [query,start_time,end_time,since_id,until_id,max_results,next_token,expansions,tweet.fields,media.fields,poll.fields,place.fields,user.fields]"}],"title":"Invalid Request","detail":"One or more parameters to your request was invalid.","type":"https://api.twitter.com/2/problems/invalid-request"}',) (400): {"errors":[{"parameters":{"pagination_token":["b26v89c19zqg8o3fosns2v373jl0fj3jxys6lrnxm8h31"]},"message":"The query parameter [pagination_token] is not one of [query,start_time,end_time,since_id,until_id,max_results,next_token,expansions,tweet.fields,media.fields,poll.fields,place.fields,user.fields]"}],"title":"Invalid Request","detail":"One or more parameters to your request was invalid.","type":"https://api.twitter.com/2/problems/invalid-request"}
geduldig commented

It appears V2 has an inconsisitency with regards to pagination. Some endpoints use pagination_token and some use next_token. I'm inquiring whether this is intended before updating the code.

geduldig commented

I got a response from Twitter. This is an inconsistency. So, I will have to amend the code.

factorlive commented

Probably, the developers of different endpoints are not talking much with each other these days. 😏

geduldig commented

It appears 2/tweets/search/recent is the only endpoint that does not support pagination_token. I'll have to build in an exception for this until Twitter fixes this.

factorlive commented

they will probably make it consistent at some point. an exception would be the best way to catch a different name of the token, indeed.

caimeng2 commented

same error message for 2/tweets/search/all

  • 👍1
geduldig commented

Will upload a fix tomorrow.

This is the change to TwitterPager.py:

                SEARCH_ENDPOINTS = ['tweets/search/recent', 'tweets/search/all']
                pagination_token = 'next_token' if self.resource in SEARCH_ENDPOINTS else 'pagination_token'
                if new_tweets:
                    self.params[pagination_token] = meta['previous_token']
                else:
                    self.params[pagination_token] = meta['next_token']
factorlive commented

With this change in TwitterPage.py, the output shows it only works within the first page and not with the pager to go beyond the first page:

{'id': '1362538499401011200', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362538321352794113', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362538308094492675', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362538285583728643', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362538143665242113', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362537885799391234', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362537838596743174', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362537789972094979', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362537408806277120', 'text': '@MattWallace888 @TwitterDev The cover photo 😩😂😂🤣'}
{'id': '1362537131718176772', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
Traceback (most recent call last):
  File "/home/patrick/icp-news/campaign/campaign/campaign.py", line 202, in <module>
    for item in pager.get_iterator(new_tweets=False):
  File "/home/patrick/icp-news/campaign/TwitterAPI/TwitterAPI/TwitterPager.py", line 123, in get_iterator
    self.params[pagination_token] = meta['next_token']
UnboundLocalError: local variable 'pagination_token' referenced before assignment
geduldig commented

Is after making the code change I suggested?

factorlive commented

yes, my comment was not clear and I revised it. Got your repo now in editable mode.

geduldig commented

What happens if you replace
pagination_token = 'next_token' if self.resource in SEARCH_ENDPOINTS else 'pagination_token'
with
pagination_token = 'pagination_token'

factorlive commented

Same problem.

Code TwitterPager.py

(...)
                else:  # VERSION 2
                    if new_tweets
                        SEARCH_ENDPOINTS = ['tweets/search/recent', 'tweets/search/all']
                        pagination_token = 'pagination_token'
                        # pagination_token = 'next_token' if self.resource in SEARCH_ENDPOINTS else 'pagination_token'
                        self.params[pagination_token] = meta['previous_token']
                    else:
                        self.params[pagination_token] = meta['next_token']
(...)

Output

{'id': '1362542072922734598', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362541974121631744', 'text': '@MattWallace888 @TwitterDev Tron has one as well trx'}
{'id': '1362541629945483267', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362541266412728331', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362540973960658951', 'text': '2021 NASCAR Cup Series DFS Power Rankings - Daytona Road Course - DFS Army https://t.co/FM2MkP2hyZ via @twitterdev'}
{'id': '1362540274031988738', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362540110475112449', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362539439227088900', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362539385019842560', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362539261250035713', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
Traceback (most recent call last):
  File "/home/patrick/icp-news/campaign/campaign/campaign.py", line 202, in <module>
    for item in pager.get_iterator(new_tweets=False):
  File "/home/patrick/icp-news/campaign/TwitterAPI/TwitterAPI/TwitterPager.py", line 124, in get_iterator
    self.params[pagination_token] = meta['next_token']
UnboundLocalError: local variable 'pagination_token' referenced before assignment
geduldig commented

I see your problem. You didn't copy code quite correctly. You must define pagination_token before the if...else. The way you have it, pagination_token is undefined for the else clause.

factorlive commented

Yep. Thanks. Pull request sent.

Got another enhancement, however, and will open another issue. Hope you don't mind 😁

geduldig commented

Fixed in v2.6.7.