candland/longlegs

A web spider that allows additional processing as the site is crawled.

A simple web spider that allows additional processing as the site is crawled.

Development

https://blog.golang.org/publishing-go-modules

Robots.txt

https://github.com/temoto/robotstxt

Deploy

https://go.dev/blog/publishing-go-modules

$ git commit -m "hello: changes for v0.1.0"
$ git tag v0.1.0
$ git push origin v0.1.0