ryyos/Amazon-scraping.PY

Hello 👋, I'm Ryo

An independent backend developer

Welcome To Amazon-Scraping.PY🛍️

This program is my training project for scraping data from Amazone's website

Feature

• The program enables to retrieve all data with electronic categories
• use logs from the logging library so that it can store request logs and response status from the API
• Use function retry to handle when getting a request timeout error
• Use of fake_useragent libraries to avoid 503 or full traffic response codes
• Use of PyQuery library for HTML parsing so that the use of selectors becomes easier
• Using the Icecream library makes it easier to debuggin

Tech

• requests is an easy-to-use Python library for interacting with APIs and making HTTP requests
• pyquery is a Python library that allows HTML and XML manipulation with a syntax similar to jQuery
• fake_useragent is a Python library that provides an easy way to generate fake user-agent strings for HTTP requests
• icecream is a Python library that provides a simple and informative way to log code, helping with monitoring program execution flows.

Requirement

• Python v3.12.0
• Pyquery v2.0.0
• icecream v2.1.3
• fake_useragent v1.4.0
• requests 2.31.0

Installation

To run this program you need to install some libraries with the command

pip install pyquery icecream fake_useragent

Example Usage

# Clone this repositories
git clone https://github.com/ryosoraa/Amazon-scraping.PY.git

# go into the directory
cd Amazon-scraping.PY

# Run code
python main.py

🚀Structure

│   LICENSE
│   main.py
│   README.md
│
├───data
│   ├───Camera_&_Photo
│   │   ├───all
│   │   └───page
│   │
│   ├───Electronics_Accessories_&_Supplies
│   │   ├───all
│   │   └───page
│   │
│   └───Results
├───libs
│   │   __init__.py
│   │
│   ├───service
│   │   │   html_parser.py
│   │   │
│   │   └───__pycache__
│   │           html_parser.cpython-312.pyc
│   │
│   ├───utils
│   │   │   logs.py
│   │   │   parser.py
│   │   │   writer.py
│   │   │
│   │   └───__pycache__
│   │
│   └───__pycache__
│
└───logs
       logging.log

Author

👤 Rio Dwi Saputra

• Twitter: @ryosora12
• Github: @ryosoraa
• LinkedIn: @rio-dwi-saputra-23560b287