A simple web crawler in Go.
Package Manager: glide
To install glide: curl https://glide.sh/get | sh
Step to run:
glide install
- run
./webcrawler
Available Endpoints:
-
Index page GET
http://127.0.0.1:8080/
-
To scrap the site POST
http://127.0.0.1:8080/crawler/
-
To view all the sitemap GET
http://127.0.0.1:8080/crawler/status/