Removal of Duplicates
Closed this issue · 2 comments
kristopolous commented
So oftentimes I noted that multiple threads would attempt to download identical images.
There will be a pull-request coming where I tried to eliminate that from happening.
kristopolous commented
Actually that doesn't do the trick because the url variable isn't thread safe. I had to use a Queue and then start the threads out of the loop. It's a significant amount of code change and so I won't open a pull request on it.
Buuut if you desire, you can find the fix here: https://github.com/kristopolous/tumblr-photo-downloader/tree/bug-003-realfix
jamiew commented
Feel free to open a new topic branch for this. I've noticed this issue too