fdrong/scrapy_celery

spider use scrapy and celery

1. introduce

a distributed crawl about using celery, scrapy, mongodb and redis

2. install

# install redis
# install mongodb
# pip install -r requirements

3. run

# cp supervisord.conf /etc/supervisord.conf
# supervisord -c /etc/supervisord.conf