If you want to get started with part 4 of the FreeCodeCamp Python Scrapy course. Follow the steps below.
This repo contains two folders:
- bookscraper (Which you can use to follow the part 4 video/article)
- full-bookscraper (The completed spider which you would have at the end of part 4)
To install the python virtual environment follow the following instructions below.
Then to activate it so that any new modules that are installed are installed into this virtual environment:
source venv/bin/activate
To install the required modules for this python project to run you need to install the required python modules using the following command:
pip install -r requirements.txt
Once the required python modules are installed you should be able to view/run the Python Scrapy Spider with the following command (from within the project folder):
View the project spiders: scrapy list
Run the project spider: scrapy crawl books
If you have issues running the pip install -r requirements.txt
command this can be due to some things not being up to date on your computer.
Running the following may solve some of these issues:
pip install --upgrade pip
The following error: NotADirectoryError: [Errno 20] Not a directory: 'pkg-config'
might be solvable by running:
export PKG_CONFIG=/path/to/pkg-config