A link scraper / website mapper I built a while back using nodejs.
Even seemingly simple websites can generate very complex routes to sources within itself. After reading about some enterprise tools, I learned that some include trackking to map how users navigate your website. Knowing what paths users take can provide great insight into many aspects of a website. A breif list of how I think such a tool could be useful includes:
- Show what links should be available at the front page (high demand resources should have the shortest routes).
- Show what links should be available from other pages (users tend to visit page B after page A, but there is no direct route) .
- Find dead links.
This (along with the fact that web-scraping seems to be a reoccuring topic) is what inspired me to start this project.