/python-spider-boilerplate

Primary LanguagePython

Python Web Spider

This project contains a simple web spider written in Python. It uses venv for environment management.

Prerequisites

  • Python 3.10.13
  • pip

Setup

  1. Clone the repository:
git clone https://github.com/blackg33/python-spider-boilerplate.git
cd python-spider-boilerplate
  1. Create a virtual environment
python3 -m venv venv
  1. Activate the virtual environment: On Unix or MacOS, run:
source venv/bin/activate

On Windows, run:

.\venv\Scripts\activate
  1. Install the required packages:
pip install -r requirements.txt

Running the Spider To run the spider, use the following command:

scrapy runspider spider.py

Replace spider.py with the name of your spider script.

Deactivating the Virtual Environment When you're done, you can deactivate the virtual environment by running:

deactivate

This will stop the shell session from using the virtual environment and return to using the system-wide Python interpreter and packages.