/Web-crawler-based-on-selenium-taking-Amazon-as-an-example

Primary LanguageJupyter Notebook

INFO 7390 Project 1: Web Data Extraction/Web Scraper

Author: QIAOTONG HUANG

Requirements:

  • Python 3.8+
  • Packages specified in requirements.txt.
  • Install with pip install -r requirements.txt.

Report and Code:

  • For detailed information about the project's ideas and process, please refer to the report.
  • To run the code:
    • Execute scrape_amazon.ipynb in Jupyter Notebook.
    • Run main.py from the code directory in the Terminal.

Code Structure:

Code
  |- scrape_amazon.ipynb
  |- main.py
  |- crawler.py
  |- export.py
  |- extractor.py
  |- settings.py
  |- amazon_login.py
  |- requirements
  |- README.txt
  |- outputs/ (All outputs are saved in this directory)