Liru/tumblr-downloader

Long time processing existing files / hard drive usage before downloading

apoapostolov opened this issue · 3 comments

apoapostolov commented

Hello Liru, I am aware you have stopped working on this little app, but if you ever change your mind, some actual database of downloaded files that speeds up each time the app is runned and analyzes content for the purpose of hardlinking, etc.

I am downloading over 200+ blogs for about 1m+ files, and each run is taking 20-30 minutes before any new content starts downloading. There's also significant hard drive usage going on, which puts damage on the hardware, although that is only side concern.

Thanks in advance!

Liru commented

I have an idea of what may be causing this; working on a fix.

apoapostolov commented

Thank you for fixing this. Can we please get a release (for Windows in particular), unless you are working on something else?

Liru commented

Was working on a proof-of-concept improvement. I sped it up to about 3 minutes of scanning time on an old netbook with 1.6 million files and a fresh cache, so it should be fine after the next commit or two. It still needs a few more tweaks.