/scrapy-broken-links

A Scrapy crawler for broken & valid links

Primary LanguagePythonApache License 2.0Apache-2.0

scrapy-broken-links

A Scrapy crawler that returns:

  • url
  • status code
  • referer

The crawler enables discovery of:

  • broken links
  • valid links