Python tool crawl websites and follow links found within the HTML. There is a config file to set domains, robot.txt obey, black listed urls/snippits, and output file.
Primary LanguagePython