Amazon Scraper

This is a amazon scraper build using selenium it can scrape product data from the website and write it to a csv file i have used selenium because it resembles a bit like human and it is a advanced testing frameworks which offers many advantages for scraping and i also used beautiful soup to extract data from the scraped data

Project Setup

1.The code is same for both windows and linux you need to install seperate chromedriver for seperate environments you can install chromedriver at https://chromedriver.chromium.org/downloads The chromedriver file should be placed at root folder
2.setup a python environment using the command python -m venv myenv
3.activate the python virtual environment for windows myenv/Scripts/activate.ps1 and for linux source myenv/bin/activate
4.install the python modules in the python virtual environment using the command pip install -r requirements.txt

Usage

simply run the project in windows python amazon_scraper.py and in linux python3 amazon_scraper.py 1.To run project in windows run the command python amazon_scraper.py 2.To run the project in linux run the command python3 amazon_scraper.py

Project Working

1.When you first run the project the selenium will scrape the product and write each product source code into html fiels in the products directory
2.In the next step the data is extracted from the html files and written to output.csv file

Project Files

1.amazon-scraper.py is the main python file which scrapes and writes data to the csv files
2.requirements.txt is the file which contains all the modules required by the project to function without errors
3.automate.bat is used to push code to github in windows
4.automate.sh is used to push code to github in linux