/web-crawl

Primary LanguagePythonMIT LicenseMIT

web-crawl

Priority

Python 2.7.*

Configure it

$ pip install scrapy

Run it

$ cd web-crawl
$ scrapy crawl products -o products.json
$ scrapy crawl details -o details.json
$ scrapy crawl brands -o brands.json
$ scrapy crawl categories -o categories.json
$ scrapy crawl supplements -o supplements.json