/pythonSimpleCrawlerBot

A Simple Crawler Bot written in Python. This script is a version of the one used at Domstry.com. You can use it to crawl the internet. It fetches domains, store the data, and adds the links found in current domain to the queue.

Primary LanguagePythonGNU General Public License v3.0GPL-3.0

Stargazers