/wget-url-crawler

A simple url-crawler for getting all urls of one domain.

Primary LanguagePythonBSD 2-Clause "Simplified" LicenseBSD-2-Clause

A very simple url-crawler for one domain based on wget and producing a csv and xlsx with url, file-extension and url segments.