In this repo we have the code that we use in our Youtube video. Feel free to clone it and play around with it
Useful commands below:
- Clone this project:
git clone https://github.com/python-scrapy-playbook/scaling-python-scrapy-redis.git
- Create a Python Virtual Environment in the project:
python3 -m venv venv
- Activate the Python Virtual Environment:
source venv/bin/activate
- Install Scrapy using pip:
pip install scrapy
- Install Scrapy using pip:
pip install scrapy-redis
- Add the urls to redis:
python3 add-urls-to-redis.py
- Running the scrapy project:
scrapy crawl scrape_quotes_worker
You can sign up here: https://redis.com/try-free/
Update the REDIS_URL in settings.py to contain your redis details.
It should look something like the following:
REDIS_URL = 'redis://<username>:<password>@<redis-connection-url>:<redis-port-number>'