Link Checker
Crawls a website searching for invalid links.
mix crawl <base_url>
Options
--max-depth
: Maximum depth the scraper will travel. Defaults to 3.
--num-workers:
Maximum amount of concurrent workers making requests to the provided URL. Defaults to 5.
Installation
In mix.exs
def deps do
[{:crawler, git: "https://github.com/mbta/404_checker"}]
end
Output
Links that respond with a status code outside the range of 200 - 399 are marked as invalid and printed.
If no links are found to be invalid, the task exits with status code 0, otherwise, exits with status code 1.