A simple script for scraping websites recursively with Node.js. Intended for use with static websites.
Built with node-website-scraper.
Edit the url, domainName, userAgent, and whitelist (for scraping external file types that are outside of a website's domain name) variables in the script.
To adjust the delay between requests, adjust the time variable in milliseconds.
The default npm start configuration in the package.json file is for executing the script in debug mode with logging and an increased memory runtime limit for the script of 5 gigabytes.
- Clone the repo
- Run the
npm installcommand - Run the
npm startcommand - Inspect the
websitesdirectory in your project during or after execution