Crawler is a Go written program designed to crawl website
- use tor SOCKS proxy to crawl hidden services
- fast, built using valyala/fasthttp (up to 10x faster than net/http)
- extract both absolute and relative URLs
- use scalable messaging protocol (nats)
- The Crawler process connect to a nats server (specified by env variable NATS_URI) and set-up a subscriber for message with tag todoSubject
- When an URL is received the crawler start crawling
- When crawling is done, the crawler will publish content to nats server with subject contentSubject and found urls with subject doneSubject