Rate limit exception raising

[fpmirabile]

I've been playing around the library. Is a great work @iSarabjitDhiman thanks for the effort on maintain it.

I have a possible suggestion (as I mentioned in other issue, I'm not the best at python)
I'm receiving the error for rate limiter, and I'd love to catch this error, I've been reading the code and it seems we are rising it, but I'm not able to capture from my end.

Is there a way we can add it? Or knowing if the rate limiter was reached?

[iSarabjitDhiman]

Yes there are several ways to get it done, I have mentioned it in #20
As people have been asking for this feature for a while now, I guess I need to add this feature to TweeterPy now.
Should it raise the Rate Limit error? Or attach the rate limit data to the data placeholder? I mean if we raise the error, the already fetched data will be lost as it won't return it. Though we can add the data to some global variable, it's not a good idea.

How about we do this :

• Add the rate limit stats to the data placeholder.
• When it returns the actual data, you can check if it has more data and if the rate limit is exhausted. This way you will know if the reason was the rate limit or there was no more data.

Feel free to share your suggestions.

[fpmirabile]

Works for me, the code in my case works soemthing like

 try:
        for user in user_payloads:
            user_tweets_payload = collector.get_user_tweets(user.id, end_cursor=last_tweet_cursor, with_replies=True, total=total_per_run)
            tweet_data = user_tweets_payload.get('data', [])
            cursor_endpoint = user_tweets_payload.get('cursor_endpoint', None)
            has_next_page = user_tweets_payload.get('has_next_page', False)
            for tweet in tweet_data:
                tweet_content = tweet.get('content', {}).get('itemContent', {}).get('tweet_results', {}).get('result', {})
                tweets_payloads.append(TwitterTweet.from_payload(tweet_content))
            
            if has_next_page and cursor_endpoint:
                update_last_processed(search_query, None, cursor_endpoint)
    except Exception as e:
        pass

(If I'm doing something wrong with the try/catch let me know)

As far as I can break that for, I'm ok with it (to avoid losing information)

[iSarabjitDhiman]

Hey I just added this rate limit feature. I am going to commit changes soon.
API rate limit stats/data will always be available in config._RATE_LIMIT_STATS , just in case you want to use it in some other way.

If you want to manually get access to this rate limits :

from tweeterpy import config
print(config._RATE_LIMIT_STATS)
NOTE : It will only show the rate limit data for the most recent request you made For instance if you recently scraped user friends, the rate limit data will be related to that API endpoint, same goes for the other endpoints.

Here is how you can implement it into your code:

from tweeterpy.util import RateLimitError

try:
        for user in user_payloads:
            user_tweets_payload = collector.get_user_tweets(user.id, end_cursor=last_tweet_cursor, with_replies=True, total=total_per_run)
            tweet_data = user_tweets_payload.get('data', [])
            cursor_endpoint = user_tweets_payload.get('cursor_endpoint', None)
            api_rate_limit = user_tweets_payload.get('api_rate_limit', {})
            limit_exhausted = api_rate_limit.get("rate_limit_exhausted")
            has_next_page = user_tweets_payload.get('has_next_page', False)
            for tweet in tweet_data:
                tweet_content = tweet.get('content', {}).get('itemContent', {}).get('tweet_results', {}).get('result', {})
                tweets_payloads.append(TwitterTweet.from_payload(tweet_content))
            if limit_exhausted:
                #raise RateLimitError('You can also pass a custom message, by default it shows "API rate limit exhausted."')
                raise RateLimitError
            if has_next_page and cursor_endpoint:
                update_last_processed(search_query, None, cursor_endpoint)
except RateLimitError:
        #handle exception here,
        pass
except Exception as e:
        pass

Other way to do it:

# if you are fetching a lot of data, avoid using while loop.
from tweeterpy import TweeterPy
from tweeterpy.util import RateLimitError


twitter = TweeterPy()
user_tweets = {"data": [],"cursor_endpoint": None, "has_next_page": True, "api_rate_limit":{}}
has_next_page = True
while has_next_page:
	try:
		tweets = twitter.get_user_tweets('elonmusk',total=50) # setting 50 on purpose, just for demonstration purpose.
		cursor_endpoint = tweets.get('cursor_endpoint', None)
		api_rate_limit = tweets.get('api_rate_limit', {})
		has_next_page = tweets.get('has_next_page', False)
		limit_exhausted = api_rate_limit.get("rate_limit_exhausted")
		user_tweets['data'].extend(tweets['data'])
		user_tweets['cursor_endpoint'] = cursor_endpoint
		user_tweets['has_next_page'] = has_next_page
		user_tweets['api_rate_limit'] = api_rate_limit
		if not has_next_page:
			break
		if has_next_page and limit_exhausted:
			raise RateLimitError
	except RateLimitError:
		#handle exception here, shuffle account or wait (time.sleep) until rate limit gets expired. Reset datetime python object is in tweets.get('reset_after_datetime_object') 
	except:
		pass

If you have any questions, let me know.

[iSarabjitDhiman]

Lets continue there.

Duplicate : #20