My name is Ilori Stephen A and I am a full-stack software developer based in West Africa, Lagos Nigeria and this project is a simple Web Scraper built in Nodejs. I have written a Blog covering this topic on Learning Dollars. You can read up on this Article there.
-
Make sure you have the latest version of Nodejs and npm installed.
-
Clone this project to your workspace directory or any directory that pleases you on your machine.
-
Open this project up in your terminal and run the command
npm installin your terminal. -
The command listed above will install the needed dependencies for this project to run.
-
You can open your
package.jsonand check under the"scripts" keyin thejson file. You will find a list of commands provided for scraping.
In this project, We experimented with Jumia: An E-commerce website in Africa that uses a Busines to Business flow. We scraped the laptop catalog under this website. You can open the app folder and edit the file BasicScraper.js if you'd like to hack.
We also experimented with webscraper.io: A website that provides some sample website for scraping. They also sell some useful plugins that could be used for scraping.
You can make a fork this project and make a pull request if you know you have some ideas you'd like to implement extending this project further. Once again, my name is Ilori Stephen A.