/crawler_demo

Primary LanguagePython

crawler_demo

  • single-thread.py -- a single thread crawler based on requests and BeautifulSoup, etc.

  • multi-thread.py -- a multiple threads crawler based on requests and BeautifulSoup, etc.

  • demo -- a crawler based on scrapy.

  demo
  ├── scrapy.cfg
  └── demo
      ├── __init__.py
      ├── items.py
      ├── pipelines.py
      ├── settings.py
      └── spiders
            ├── __init__.py
            └── example.py