A simple web crawler written in Go.
go install github.com/aifaniyi/crawlercrawler https://sample.url.com/pathThe crawler can be stoped using ctrl + C
- Stop crawler once there are no more urls to visit.
- Add timeout on http client to prevent client blocking by unavailable sites.
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
Please make sure to update tests as appropriate.