Spider

An aggressive, multi-threaded spider for discovering all reachable HTTP endpoints on a host.

Usage

Some sites or WAFs may block this tool for sending too many requests too quickly. A mitigation strategy is to use the -frequency flag to limit the rate of requests.
Side effect free functions are unit tested but integration tests would also be useful if this was to be deployed in production.
The current status indication features are not great. There's a debug output that shows requests being made and any errors, and results are printed after crawling is finished. A better way would be to refresh the screen with information such as links visited, links in queue, unique links discovered, etc. Also, a sitemap could be generated instead of links being printed out.