A script that crawls every anchor tags (a) that exists on a given URL.
python -m venv env
cd env && source bin/activate
git clone https://github.com/sosolidkk/url-crawler.git
cd url-crawler
pip install -r requirements.txt
$ python run.py -u https://scrapethissite.com/ -f file_name.json
$ python run.py --help
- Fork it (https://github.com/your-github-user/url-crawler/fork)
- Create your feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Add some feature'
) - Push to the branch (
git push origin my-new-feature
) - Create a new Pull Request
- sosolidkk - creator and maintainer