jjjake/internetarchive

Upload: Retry on exceptions instead of crashing

Opened this issue · 2 comments

imerr commented

Hi
I've been trying to upload some things to IA, unfortunately I would come back to the ia upload command having crashed due to connection timeouts very often
It'd be good if those would be handled more gracefully and just retried unless there's a reason they are not currently?
I have patched this in for myself, but that's more of a duct tape fix to get things working (infinite retries with 1s sleep inbetween on exception) and I figured I'd reach out for feedback how this should be handled properly

I see there is already a retry flag, but that only applies to 503 responses. I would suggest to expand that to also include connection errors and the like? Although personally I'd like it to just retry indefinitely on network errors (with proper backoff between attempts?), so maybe expanding the flag with a special value to do that would be good (but that's an issue for another time)

Let me know what the preferred way of handling this is and I'm happy to get that coded out and submit a pr (or if someone else wants to that's fine too)

maxz commented

Exponential backoff. Also it should probably only retry if the server responded with a "100 Continue" for the same action before.

I don't think 100s are relevant; requests doesn't support Expect: 100-continue anyway.