/website-scraper-puppeteer

Plugin for website-scraper which returns html for dynamic websites using puppeteer

Primary LanguageJavaScriptMIT LicenseMIT

Version Downloads Build Status

website-scraper-puppeteer

Plugin for website-scraper which returns html for dynamic websites using puppeteer

Requirements

  • nodejs version >= 8
  • website-scraper version >= 4

Installation

npm install website-scraper website-scraper-puppeteer

Usage

const scrape = require('website-scraper');
const PuppeteerPlugin = require('website-scraper-puppeteer');

scrape({
    urls: ['https://www.instagram.com/gopro/'],
    directory: '/path/to/save',
    plugins: [ new PuppeteerPlugin() ]
});

How it works

It starts Chromium in headless mode which just opens page and waits until page is loaded. It is far from ideal because probably you need to wait until some resource is loaded or click some button or log in. Currently this module doesn't support such functionality.