Add exponential backoff as an option when creating the client.
Opened this issue · 3 comments
If you fire off a lot of requests at one time it is possible to go over the 100 requests in 100 seconds limit. Once it hits that limit datasheets throws an error. It should be possible to have datasheets retry by default using something like exponential backoff.
Hey Ricky, greetings from Ethiopia. :)
That makes sense to me and seems reasonable. Any interest in taking a shot at implementing this?
A quick search makes it look like this isn't built into the Google Sheets or Google Drive APIs, which I'm not surprised by. My thought is that we could have a decorator that does a try-except for the appropriate errors and is smart enough to keep track of how much more to back off.
Also, we likely want a maximum back off, at a minimum for non-service-account clients. The reasoning there is that the typical use case of non-service-account access is interactive use, and if the issue is something like internet connectivity being down then it'd be better to give quick feedback than have the user's code hang forever. I guess there should be an upper limit for service account use as well, though maybe it could be higher.
What do you think?
I will try to take a shot at it before I leave squarespace. I think the decorator is a great solution. Only problem right now is it should probably be behind the auth library change as the error will change when we change packages.
Maximum backoff sounds like a better idea. The quicker feedback would be useful. Service accounts should have a higher threashold enabling them to be more resilient as they are usually used in automation.
__getattribute__ might be a better place for it then a decorator. It will let us control exactly when it is called.