soniCrawl

Multi-processed and multi-threaded python web crawler with mongoDB backend with mongoDB cache