🕸 🕷 scrapy-bootstrap 🕷 🕸
Basic configuration to start with your next web crawling project.
Get Started
# install virtualenv to isolate the project dependencies
pip3 install virtualenv
# init the virtualenv
virtualenv venv
# activate the virtualenv
. ./venv/bin/activate
# install the project dependencies
pip install -r requirements.txt
Usage
To initiate any of the spiders and export in CSV use:
scrapy runspider spiderweb/spiders/sample_spider.py -t csv -o results.csv