Finding a bike is complicated... This project helps you by :
- tracking the prices of your favorites bikes
- storing them in a mongo database
For every website, you will have to create your own scraper. Each scraper has two main methods :
parse_bs
: usesBeautifulSoup
to parse items. It's the simplest oneparse_selenium
: usesselenium
to parse items. It's more complicated but also more interesting ! For example, you have the possibility to navigate on your web pages and click on dropdowns to see if your frame size is available
To install this project, you only need to have Docker installed and run the build command :
docker-compose build
Please look at the config.py
file and modify the mongo parameters.
To add your favorite bike :
- Add it to the
SCRAP_DICT
dictionary, with a name, a url, and a size if you want to. - Create a new scraper class in the
scrapers.py
file and adjust it to your needs. - Don't forget to complete the
build_scrapers
function ;)
The project runs two containers :
- the mongo server
- the web scraper
Launch everything with the following command :
docker-compose up
Enjoy and be patient !
INFO: You can use MongoDB Compass to analyze your database.
A lot of things !
- add a
dash
dashboard - handle the cookie popups that block the navigation with
selenium
- send notification if your frame size is available or if the price is decreasing