jamiew/tumblr-photo-downloader

Removal of Duplicates

Closed this issue · 2 comments

So oftentimes I noted that multiple threads would attempt to download identical images.

There will be a pull-request coming where I tried to eliminate that from happening.

Actually that doesn't do the trick because the url variable isn't thread safe. I had to use a Queue and then start the threads out of the loop. It's a significant amount of code change and so I won't open a pull request on it.

Buuut if you desire, you can find the fix here: https://github.com/kristopolous/tumblr-photo-downloader/tree/bug-003-realfix

Feel free to open a new topic branch for this. I've noticed this issue too