/build-your-web-scraper-full-js-sosnoob

Scrap a website data & Build your own Database / API for your apps more free tuto on sosnoob.com

Primary LanguageJavaScriptApache License 2.0Apache-2.0

License Awesome

SosNoob - Web scraper (full JS) - get data & build your own API / Database

This repository is the complete code for the tutorial on SosNoob.com : "How te build your own web scraper easily?" You need a web site data to build a database or an API with it ? This tuto is for you !

The program can be export as an ".exe" (so don't need to have any dependency installed) and generate your data into JSON files. You can also modify it to directly push data on your database / API. For this example I scrap Github's data, especially the result list of "Javascript" in Github's search area.

The program use pkg dependency which allow you to export it as an executable for :

  • Windows
  • OSX
  • Linux

So you or your community don't need any dependency once it export (no node js).

Download : All versions are available here

⚠️ Don't use the tool for more than 20 pages per hour in order to ovoid any IP ban from destination web site.

This program was built to be very light full, it can run on almost every configurations.

In fact SosNoob scaper is a tool :

  • 100% performance : Developed to be very light full
  • Simple to use | Plug and use

JSON format

An item result example

{  
  "name":"item name",
  "url":"Item's link",
  "labels":[ "label 1", "label 2", ["..."], "label n" ]
}

Technologies choice

Sosnoob scraper is build with full javascript : NodeJs/Npm, javascript.

Bug & Crash | Proposal ?

Don't hesitate to create an issue or ask a question on sosnoob.com comment thread.

Discord

Coming soon !