sobne/webcrawler

webcrawler is an open source web crawler written in .Net