Auto-resume of uploads until uploads are fully uploaded?

[GoogleCodeExporter]

Describe the steps that will reproduce the problem?

1. Upload large files > 500-1000mb
2. Not great connection (upload max 50-60kb/s)
3. No limit rate, connection sometimes drop and I get "pycurl.error: (56, 
'Failure when receiving data from the peer') in c.perform()" on line 149 in 
/usr/../bin/youtube-upload

Versions:

youtube-upload: 0.7.0
python: 2.6
python-gdata: 2.0.14

Please provide any additional information you deem to be useful:

The problem is just a general problem of youtube-upload not able to handle a 
temporary connection error with pycurl and hence failing to resume 
automatically if a small temporary connection error occurs during uploads that 
can last many hours. Thus the need to restart the upload again everytime it 
fails, which is annoying for very large files. 

Adding:

    c.setopt(c.RESUME_FROM, sum(os.path.getsize(path) for path in files_params.values()))
    c.setopt(c.MAX_SEND_SPEED_LARGE, 51200)

on line 130/131 in youtube-upload seems to clear my problem, however I don't 
know if that is the right way? as c.RESUME_FROM sets the continuation bit, but 
does that mean curl autoresumes until the upload is over?

Original issue reported on code.google.com by solarphe...@gmail.com on 22 Sep 2011 at 1:46

[GoogleCodeExporter]

> c.setopt(c.RESUME_FROM, sum(os.path.getsize(path) for path in 
files_params.values()))

this works? I don't get it, this makes sense for downloads, but for uploads 
it's setting the continuation point to the sum of complete file sizes...

> c.setopt(c.MAX_SEND_SPEED_LARGE, 51200)

Since you have to tweak curl options to get reliable uploads, you better use 
--get-upload-form-info + curl (see the example in the sources). I know it would 
be better to set pycurl options, but other than -o "KEY=VALUE" (which means the 
user needs to know about low-level curl options) I don't see how to do it.

Original comment by tokland on 22 Sep 2011 at 1:56

[GoogleCodeExporter]

Thanks, I tried the --get-upload-form-info + curl, it works, but for my use 
it's not great as I have:

youtube-upload --email=... --password=... --title=... --description=... etc. >> 
outputfile
so that it happens the youtube video url easily to file afterwards, with 
--get-upload-form-info + curl I couldn't be bothered to see how to get the 
video url easily...

Anyway, I don't understand why you did not just add a rate_limit option to

> def post(url, files_params, extra_params, rate_limit=False, 
show_progressbar=True):
and add something like:

> if rate_limit != False & is_integer( rate_limit )
>    c.setopt(c.MAX_SEND_SPEED_LARGE, rate_limit)

** I don't know the python syntax exactly, I don't think that is_integer() 
function exists, but you understand what I mean :p **

 and add an option --rate-limit to youtube-upload itself that's not very difficult and thus skip the need for --get-upload-form-info + curl.

Just suggestion ;)
Is there something more beneficial to using curl outside of youtube-upload? 
from what I see youtube-upload uses curl anyway, except if --api-upload is 
specified.

As for my problem, I thought also the 
> c.setopt(c.RESUME_FROM, sum(os.path.getsize(path) for path in 
files_params.values()))
made no sense, as for the upload the file on my computer is always the same 
size, I would need to find at which size the upload stopped... But I don't know 
how to get that information out of curl when it throws an error?

In the past I used curl to upload other files directly in bash with a command 
like that:

curl -T FILE FTP_URL_TO_UPLOAD_TO --user username:password ||
        while ! curl -C- -T FILE FTP_URL_TO_UPLOAD_TO --user username:password ; do true ; done

That would auto-resume the upload until it was complete, that's what I was 
thinking would be good to implement in youtube-upload. It was for FTP but I 
guess the same can be done for HTTP post uploads too?

I have another 10gb of files to upload, so I'll try different modifications in 
youtube-upload and keep you updated, I think it's the fact that I limited the 
rate that prevents my connection from ever failing, but I'm not sure, next time 
I'll try just with:

> c.setopt(c.RESUME_FROM, sum(os.path.getsize(path) for path in 
files_params.values()))

and next just with:

> c.setopt(c.MAX_SEND_SPEED_LARGE, 51200)

to see which one of the two is actually making my uploads work..

Original comment by solarphe...@gmail.com on 22 Sep 2011 at 3:55

[GoogleCodeExporter]

The script has already lots of options, that's why am I relucant to add more. 
Of course adding an option to limit upload speed is easy, but as you know curl 
has hundred of options... once you start, where to stop? 

I'll think about it, but first try if really a speed limit is all you need.

Original comment by tokland on 22 Sep 2011 at 4:17

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

I let the upload run the whole day and sadly the speed limit or c.RESUME_FROM 
changes nothing, it seems it's in the evening that my internet connection is 
not as reliable as during the day. It's always the same error that is thrown:
pycurl.error: (56, 'Failure when receiving data from the peer')
My connection just fails for some seconds once in a while, the router goes down 
because of 'LCP down' :(

So implementing auto resumable downloads is the only to make it safe for 
uploads on my connection it seems, I had a look at
http://code.google.com/apis/youtube/2.0/developers_guide_protocol_resumable_uplo
ads.html
it doesn't seem to difficult to modify youtube-upload to catch that error and 
resend a post to get the failed upload size and resume

Original comment by solarphe...@gmail.com on 22 Sep 2011 at 8:32

[GoogleCodeExporter]

AFAIK python-gdata uses youtube api 1.0, am I wrong?

Original comment by tokland on 22 Sep 2011 at 8:37

[GoogleCodeExporter]

bump, I don't know, I just started with all that 2 days ago, all I saw is
http://code.google.com/apis/youtube/2.0/developers_guide_protocol.html#API_Versi
oning
you can set GData-Version to use of api 2, by default it's 1 that's used

Original comment by solarphe...@gmail.com on 22 Sep 2011 at 9:00

[GoogleCodeExporter]

Yeah, but that's the generic API, the python-gdata library implements v1:

http://code.google.com/intl/es/apis/youtube/1.0/developers_guide_python.html

Original comment by tokland on 22 Sep 2011 at 9:13

[GoogleCodeExporter]

Hmm, so I guess I it's stalemate, i'll have to make my own script to upload is 
I want resumable uploads :(
Thanks for the info

Original comment by solarphe...@gmail.com on 22 Sep 2011 at 9:46

[GoogleCodeExporter]

I am afraid so. It's really weird Google has not yet updated python-gdata to 
v2, the API has been out there for quite a while now.

Original comment by tokland on 22 Sep 2011 at 10:15

• Changed state: WontFix

[GoogleCodeExporter]

Original comment by tokland on 13 May 2014 at 2:55