tiny-scraper

a simple web scraper, friendly usage.

Feature

request will be queued, configurable request frequency and delay.
page parse logic can be customed base on url route.

Dependencies

flyd
transducers.js
path-to-regexp
co

Install

npm install tiny-scraper

API

createRouter

return a router to parse specified page.

const { createRouter } = require('tiny-router');
const router = createRouter();

router.match

match a site base uri, return a function to filter urls in this site. please refer to path-to-regexp document for route expression format.

Parameters

baseUri

const matchGithub = router.match('https://github.com')

matchGithub(
  '/zhangmq/tiny-scraper',                //route expression
  function* (req, res, params, query) {
    yield storage(res.data)               //storage just for demo, you can implement it by yourself.
    return [/* parsed urls */];
  }
);

createScraper

create a scraper.

Parameters

options a object contains config fields.
- maxRequest max requests count paralleled.
- requestDuration min request duration, if request completed early, will wait until specified duration.
- router you implemented router.
- downloader method to request page, config => responsePromise. example: axios.request

const { createScraper } = require('tiny-scraper');
const scraper = createScraper({
  maxRequest: 1,
  requestDuration: 2000,
  router,
  downloader: axios.request
});

scraper.tasks$([/* seed tasks */])

scraper.task$

task input stream. you can send seed url or resend failed request into this steam.

Parameters

input a array of request config. please refer to axios document.

scraper.running$

current running tasks.

scraper.requestError$

failed request stream.

scraper.routeError$

route execute error. you can debug you route code by this scream.

Demo

example