- Install python3, pip3, and libmagic for your OS.
Yum, APT, and Homebrew all work well for this. 2. Set up your python dependencies.
virtualenv -p python3 venv
source venv/bin/activate
pip3 install -r requirements.txt
- ...
- Profit (or, at least employment).
python3 crawler.py --help
- Only uses file extensions (.txt, .zip)
- Only goes into .zip and not .tgz/.tar.gz
- No error handling... shame on me