An e-commerce scraper born from the love of software development and videogames. This project was initially created because every time I wanted to buy a videogame, I had to browse on several pages to find:
- Stores with the best prices.
- Stores that had stock of the game.
- Stores that were selling the game on pre-order.
- Stores that had a discount.
- If a game price dropped.
- Etc.
And so I created this tool to gather the products information into a centralized database, and then with a simple search I could get all what I was looking for.
The project is divided into multiple packages:
packages/scraper
The most important part of the project. It's a Node.js script that gathers the information across multiple websites using the scraping method to populate a database.
Tech stack: Node.js, Typescript, Axios and Cheerio.
packages/database
A Node.js library to serve as an abstraction layer to communicate with the database.
Tech stack: Typescript, Knex and PostgreSQL.
packages/ui
A React library with shared components and styles. It works as a system design across the project websites.
Tech stack: React, Typescript, Storybook and SASS.
packages/utils
A Node.js library with shared utilities.
Tech stack: React and Typescript.
packages/website
A Next.js website to search for the scraped information. It fetches, filters and sorts the products on the server (supports server-side rendering).
Tech stack: Next.js, React, Typescript, Redux and SASS.
Make sure you have installed all of the following prerequisites on your machine:
- Git
- Node.js
- Docker
- Yarn - install Yarn globally using npm:
npm install --global yarn
- Clone the repository
git clone https://github.com/cristian-azocar/ecommerce-scraper.git
- Install the dependencies
cd ecommerce-scraper
yarn install
- Build the project
yarn build
- Build and start the Docker container
docker-compose up --build
- Create and populate the database
yarn database migrate
yarn database seed
If not already running, start the container:
docker-compose up -d
To execute the scraper:
yarn scraper start
To start the website:
yarn website dev
To start Storybook:
yarn ui start