/awesome

a list of some interesting repositories, tools

Awesome

Scrapy Distributed

  • crawlab - 基于Golang的分布式爬虫管理平台
  • Gerapy - Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js
  • scrapydweb - ScrapydWeb: Web app for Scrapyd cluster management

Scrapy Middleware

command line

  • cleo - Cleo allows you to create beautiful and testable command-line interfaces.

HTML parser

  • scrapely - A pure-python HTML screen-scraping library

Crawler

Tools

  • qrcode - Python 艺术二维码生成器

utils

  • queuelib - Collection of persistent (disk-based) queues