/node-spider-man

The spider to grab the future of the internet using nodejs. Defined patterns, then get data

Primary LanguageJavaScript

node-spider-man

NPM version Downloads License

Features

  • Transform the page into JSON format response according to the rules
  • Use fetchQueue to manage the fetchRequests in the order
  • Support sync and async mode.
    • sync: Execute each fetchTask after the previous one finished
    • async: Execute each fetchTask after the delayFetch time
  • Support queue finished callback
  • Support to PAUSE or CONTINUE the spider's work(>=0.3.0)

How to use

npm install node-spider-man --save

var spiderMan = require('node-spider-man');

Demo

The demo shows the experiement that we can grab what we want from the internet using node-spider-man

Lisence @Apache-2.0

Copyright to @2012-2016 Staydan.com