/Simple-Crawler

Simple web crawler with given sites

Primary LanguagePythonGNU General Public License v2.0GPL-2.0

Simple-Crawler

Simple web crawler that crawl given sites at first and store them in hard drive, next search inside them to find your prefered keyword(s). In the next phase it will be able to parse the downloaded sites and return suitable .pdf file include text and links.

#Usage create targets.txt and sites.txt files in main directory,write your preferred sites in sites.txt and your keywords in targets.txt. then:

python main.py