HTTPError quotaExceeded No documentation for handling pagination without generator
Closed this issue · 3 comments
Fetching comments to videos from YouTube data API sometimes I can get over million of them, which will always consume all of my daily quota.
In a situation like this I'm getting HTTPError 403 quotaExceeded error, and after caching raised exception, generator loses its internal state (It's python's generator thing).
I'm looking for some way to continue fetching comments from place that thrown the exception after quota reset.
Interesting issue. I just reread my old code and it seems that you're looking for the next_page method. Contrary to one might expect, the pages aren't tracked by page numbers. Rather a random token.
For guidance read from here until line 320.
In your code you will need to do sth very similar to the above + add a try except clause to catch the last response before the 403 error you're encountering.
A bit unrelated, have you considered using partial requests? Or batch requests (not implemented by this lib)? They can give you more mileage before you hit a QuotaExceededError.
My solution
I solved it like this:
https://gist.github.com/Vergenter/fa0c4710e6c4adabcd293f270333402b
Unrelated
Partial requests:
From update https://developers.google.com/youtube/v3/revision_history#july-29,-2020
We have simplified our process for charging quota for API requests by removing the additional cost associated with the part parameter. Effective immediately, we will only charge the base cost for the method that is called.
It means that there is no benefit from partial requests in terms of quota.
Batch requests:
As explained in https://stackoverflow.com/questions/63721059/do-batch-requests-for-the-youtube-data-api-work , You can only batch request using field maxResults, so I think there is no more ways to optimize quota usage.
Hidden quota:
Additional problems come from "hidden quota". Sometime from requests I get quotaExceeded error even if I have quota left, and then I need to wait for some time(like 1 hour) to use remaining quota for this day.
Awesome, thanks for the detailed explanation!