abratnap/ray-crawler

A distributed web crawl engine based on Ray.