A simple crawler that uses the Breadth First Search algorithm to crawl a given website and its sup-pages. It reports any links that lead to websites that are not available (i.e. 'broken links'). I wrote a similar crawler around 4 years ago using PHP. I added it to the repository for comparison.
PJUllrich/Link-Crawler
Python Crawler that reports broken links on a given website and its sup-pages
Python