/url-crawler

Primary LanguagePythonMIT LicenseMIT

url-crawler

A script that crawls every anchor tags (a) that exists on a given URL.

Installation

  1. python -m venv env
  2. cd env && source bin/activate
  3. git clone https://github.com/sosolidkk/url-crawler.git
  4. cd url-crawler
  5. pip install -r requirements.txt

Usage

$ python run.py -u https://scrapethissite.com/ -f file_name.json

Info

$ python run.py --help

Contributing

  1. Fork it (https://github.com/your-github-user/url-crawler/fork)
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create a new Pull Request

Contributors