/Douban-House

Crawling house information on Douban using proxy pool

Primary LanguagePythonMIT LicenseMIT

Douban House Crawler: A Proxy Pool Method

Use proxy ip pool to tackle anti-spider of Douban

Project structure

image-20220616124004493

Main dependencies

  • requests
  • MySQL

How to run

  • 🍺 [Env] Run pip install -r requirments.txt to install the env
  • 🍺 [DB] In MySQL console, run source houses.sql
  • 🍺 [Custom] Modify conditions in config.py
  • 🍺 [Proxy Pool] clone this repo, and run docker-compose up
  • 🚀 [Run] python main.py