/Shopclues-Crawler

A crawler done using Python, Selenium and Beautiful Soup

Primary LanguagePython

API Call

To run this project

Install a Webdriver

Install a Webdriver for Selenium based on the browser you are using in my project I have used chrome webdriver. Ensure that you store it in C drive. For example: "C:\Program Files (x86)\" --> This is where I have stored my webdriver

Data storage

To store your data which you have generated, generate an API key for google sheets and google drive you will have the json file with you. Refer this video if you face difficulty. This is the thing which I did. Link: https://www.youtube.com/watch?v=PyaRgFJBnH4&pp=ygUtaG93IHRvIHB1c2ggZGF0YSB0byBnb29nbGUgc2hlZXRzIGZyb20gcHl0aG9u

Webdriver

While running your code ensure that you monitor both the webdrivers which are there since sometimes due to continued usage one of them might fail to reload also keep scrolling the second webdriver window till the bottom for the button. Sometimes due to system glitches it may not reload properly.

Database

While pushing data to database ensure that you won't overload the database with the data scrapped.