teracow/googliser

incomplete images are ending up in the gallery

Closed this issue · 5 comments

Edited title as I'm not yet sure why this is happening.

Test parameters were: -p cows -DSA -P200 -n1000

It's looking like there may be a bug in the way Wget retries downloads. It's causing the retried file to be appended to the original incomplete file.

Just found this on the Wget bugtracker: http://savannah.gnu.org/bugs/index.php?51840

Looks like the same problem. :(

Unable to fix this. Wget needs fixing instead. :(

Well why not just rely on curl until the bug is fixed with wget?

Well why not just rely on curl until the bug is fixed with wget?

counter-offer: maybe we should prefer curl, only using wget as a fall-back (for systems without curl)? This would reduce the likelyhood of this bug messing-up downloaded images.