Custom Built Scrapping Scripts
# install dependencies
$ npm install
# serve with hot reload at localhost:3000
$ npm run dev
# build for production and launch server
$ npm run build
$ npm run start
# generate static project
$ npm run generate
- SCRAPPER SETUP
- SCRAPPER OPTIONS
- DATABASE CONNECTION
- import puppeteer
const puppeteer = require('puppeteer');
- Start an async function:
const functionName = async () => { }
- Create a base URL variable
const baseUrl = 'https//www.example.com'
- Create a broser variable
const browser = await puppeteer.launch({ headless: true })
- Create a page variable
const pager = await browser.newPage()
- Use the page variable to vist the baseURL
await page.goto(baseUrl, { waitUntil: 'newworkidle2' })
- Calculations are performed inside a
evaluate
function:const scrapedData = await page.evaluate(() => { })
- Close tht browser
await browser.close()
Connection String: mongo "mongodb+srv://to-do-list-s6ku5.mongodb.net/test" --username <username>