/urlcrawl

url crawler

Primary LanguageGoMIT LicenseMIT

URL Crawl

License Coverage GitHub Workflow Status Go Report Card

Get a list of URLs from a given URL.

Download data to destination directory.
If file already exists, skip download again. URI if different from the given URL will be skipped.

Usage

urlcrawl -d "./testdata" <url>