darwinz/js-cli-web-crawlers

Some web crawlers run through CLI, developed in JavaScript

JS CLI Web Crawlers

Prerequisites

• Node.js 8.5+
• Npm 5.3+

Setup

After all prerequisites are met, install package dependencies with npm

$ npm install

Then run the config helper to configure the environment (prompts for user configuration options)

$ npm run config:helper

Running

After all prerequisites are met and setup is complete, crawlers can be run from shell using node, or as a node web service

Example (running from cli)

$ node crawlers/vivint_solar --run

or

$ npm run crawlers:solar

Example (running as web service)

$ npm start

$ curl -XGET http://localhost:3000/crawlers/solar/production

Crawlers

• Vivint Solar - check production output
• Google Analytics - check real time analytics data

Testing

Integration testing can be run using npm

$ npm test