FR: --dry-run feature
Zocker1999NET opened this issue · 2 comments
If I did not miss anything, PFERD as of now has no feature to test / try out what files a crawler would download (and how it would save those) without actually downloading them. I would find this very helpful so while creating my configuration file, I can re-assure (to some extend) that PFERD does not fill up my drive with a bunch of files I did not account for.
After a successful crawl, you can use --debug-transforms to test changes to your transforms. While this is not a full dry run, it can help in some cases.
Thats good to know, but I have already seen that.
I was more interested into using that dry-run feature before downloading a whole new course and especially to avoid producing unnecessary load & using up too much storage when e.g. the course contains unexpectedly big files or (which I have not seen yet) just in case PFERD misclassifies video files and downloads then anyway while having video=no set.