Scrapper to crawl UK and US Lego Retiring Soon pages
Table of Contents
Live Demo: https://peaceful-dawn-14209.herokuapp.com/
LEGO is a very popular toy company. They have a huge number of products and occasionally decide to stop producing some of these products. The LEGO website has a list of products that are no longer in production and will soon be unavailable for purchase. They describe these products as ‘retiring soon’.
We’d like to build a tool to monitor when products are added to the list of products soon to be retired. Our idea is that as soon as a product is added to the list, we will purchase a number of the units available, and then sell them in the future for a higher price when the product becomes harder for customers to purchase.
This is a simple tool that will monitor the LEGO website and record when new products are added to the "Retiring Soon" page.
This will involve scraping specific URLs on a regular basis, reporting the data with a simple UI, and sending email alerts.
It collects the following attributes
- Marketplace (UK or US)
- Product name
- Product Price
- Sale Price
- Discount amount
- Discount percentage
- Product details page URL
- Stock status (Available/Out of stock)
After performing an update, the tool can send an email alert (to the email stored in MAIL_RECIPEINT environment variable) with a summary of any new products discovered, removal or updation of existing products.
This section should list any major frameworks that you built your project using. Leave any add-ons/plugins for the acknowledgements section. Here are a few examples.
To get a local copy up and running follow these simple example steps.
This is an example of how to list things you need to use the software and how to install them.
- npm
npm install npm@latest -g
- Install composer
- Clone the repo
git clone https://github.com/your_username_/Project-Name.git
- Install NPM packages
npm install
- Install composer packages
composer install
- Run migrations
php artisan migrate
- Boot the local server
php artisan serve
- Serve static assets locally using webpack (different terminal)
npm run watch
To run the scraper manually, you can use the following command
php artisan scrape:legoYou can specify the market using the market option. (Available options: uk, us)
php artisan scrape:lego --market=usIf you want to scrape website without receiving email updates use the silent option
php artisan scrape:lego --market=us --silentDistributed under the MIT License. See LICENSE for more information.
Nishat Sayyed - LinkedIn
Email - nishatsayyed26@gmail.com
Project Link: https://github.com/your_username/repo_name