A Python scraper built with the Scrapy framework to extract classified ads listings from www.click.in. The scraper utilizes proxies to avoid bans, rotating proxy middleware for IP rotation, and output processors for data cleaning.
- Scrapes classified ads listings from www.click.in.
- Utilizes proxies to avoid bans.
- Implements rotating proxy middleware for IP rotation.
- Uses output processors for data cleaning.
-
Clone the repository: git clone https://github.com/Eben001/classifieds.git cd classifieds install a virtual environment if you wish
-
Install dependencies:
pip install -r requirements.txt
Configuring Proxies
- Obtain Proxies: Acquire a list of proxies from a reliable provider.
- Update proxies.txt with the proxies
- Run the Scraper:
scrapy crawl clickin
- To output the result in a json file run:
scrapy crawl clickin -o output.json
Feel free to customize the content based on additional features or specific instructions for your project.