'handle_rate_limit' incorrect behaviour
Opened this issue · 1 comments
Description:
When creating an instance of Client
(found in twitter_ads.client
) you can pass a dict with additional options, using the options
parameter. One of this options is handle_rate_limit
. When passed, the library is supposed to look for the rate limit headers returned by the API and sleep the execution when such a limit is encountered. However, it would appear that the wait time is insufficient, as the next request after the wait period fails with a rate limit error.
How to replicate:
client = Client(**config, options={'handle_rate_limit': True})
...
for chunk_ids in split_list(ids, 20):
analytics_data: List[dict] = Campaign.all_stats(
account,
chunk_ids,
[METRIC_GROUP.ENGAGEMENT, METRIC_GROUP.BILLING],
granularity=GRANULARITY.TOTAL,
start_time=export_range.start_date,
end_time=export_range.end_date
)
The above code is ran for an export range of one day. It is repeated for multiple accounts. This is the output when a rate limit is reached:
time=2022-07-26 17:22:50 name=twitter_ads.http level=WARNING message=Request reached Rate Limit: resume in 592 seconds
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "*********/twitter-ads-env/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "*********/twitter-ads-env/lib/python3.8/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "*********/twitter-ads-env/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "*********/twitter-ads-env/lib/python3.8/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "*********/utils/backfill.py", line 110, in run_backfill
for entity_data in results:
File "*********/utils/backfill.py", line 74, in _download_data
analytics_data: List[dict] = Campaign.all_stats(
File "*********/twitter-ads-env/lib/python3.8/site-packages/twitter_ads/analytics.py", line 78, in all_stats
response = Request(account.client, 'get', resource, params=params).perform()
File "*********/twitter-ads-env/lib/python3.8/site-packages/twitter_ads/http.py", line 70, in perform
raise Error.from_response(response)
twitter_ads.error.RateLimit: <RateLimit object at 0x7f2b79a75520 code=429 details=None>
Expected behaviour:
The library should wait the appropriate amount of time before attempting to call the API again to completely avoid RateLimit
errors.
Observed behavior:
At the current state, the handle_rate_limit
option serves no purpose. Despite waiting a certain amount of time, the rate limit error is encountered anyway.
@pep4eto1211 i believe this is this related to #298. can you confirm?