- Install dependencies:
pip install -r requirements.txt
- Update
START_PAGE = 'https://www.scrapebay.com'
in broken_links/spiders/find_broken.py - Run the spider:
scrapy crawl find_broken -o output_file.csv
pip install -r requirements.txt
START_PAGE = 'https://www.scrapebay.com'
in broken_links/spiders/find_broken.pyscrapy crawl find_broken -o output_file.csv