micromanchg

micromanchg's Stars

• nicodjimenez/bus_catchers

  Python scripts for scraping bus ticket data from the websites of BoltBus, Greyhound, Megabus, GoBus, Amtrak, Peterpan, and EasternTravel.

  Language:Python3923

• fabienvauchelles/scrapoxy

  Scrapoxy is a super proxy aggregator, allowing you to manage all proxies in one place 🎯, rather than spreading it across multiple scrapers 🕸️. It also smartly handles traffic routing 🔀 to minimize bans and increase success rates 🚀.

  Language:TypeScript2k233

• aivarsk/scrapy-proxies

  Random proxy middleware for Scrapy

  Language:Python1.7k409

• kohn/HttpProxyMiddleware

  A middleware for scrapy. Used to change HTTP proxy from time to time.

  Language:Python327132

• cocoakekeyu/autoproxy

  一个用于scrapy爬虫的自动代理中间件

  Language:Python14830

• scrapy/scrapyd

  A service daemon to run Scrapy spiders

  Language:Python2.9k571

• BruceDone/awesome-crawler

  A collection of awesome web crawler,spider in different languages

  6.4k707

• DormyMo/SpiderKeeper

  admin ui for scrapy/open source scrapinghub

  Language:Python2.7k507

• ferventdesert/Hawk

  visualized crawler & ETL IDE written with C#/WPF

  Language:C#3.2k974

• tigerxue/ghost-login

  Language:Java2813

• tigerxue/SeimiCrawler

  一个敏捷的，分布式的爬虫框架;An agile, distributed crawler framework.

  Language:Java5

• kakaok/scrapy_multi_pipeline

  Language:Python137

• ioiogoo/scrapy-monitor

  scrapy-monitor，实现爬虫可视化，监控实时状态

  Language:Python10834

• happyte/buyhouse

  🏠基于python的scrapy爬虫，爬取链家网成都地区新房源，并用高德api在地图上可视化显示

  Language:HTML10743

• xtt129/seCrawler

  A scrapy project can crawl search result of Google/Bing/Baidu

  Language:Python7641

• LKI/wescraper

  依赖Scrapy和搜狗搜索微信公众号文章

  Language:Python4627

• xiaodaguan/sogou_weixin

  weixin.sogou.com 微信爬虫 -- 基于scrapy

  Language:Python2831

• yinzishao/NewsScrapy

  基于scrapy的新闻爬虫

  Language:Python9432

• fankcoder/findtrip

  机票爬虫（去哪儿和携程网）。flight tickets multiple webspider.(scrapy + selenium + phantomjs + mongodb)

  Language:Python440252

• hk029/LagouSpider

  【图文详解】scrapy爬虫与动态页面——爬取拉勾网职位信息（1）

  Language:Python8150

• hailong0707-zz/spider_news_all

  Scrapy Spider for 各种新闻网站

  Language:Python10554

• LiuXingMing/SinaSpider

  新浪微博爬虫（Scrapy、Redis）

  Language:Python3.3k1.5k

• lihansunbai/Fang_Scrapy

  这是一个作者毕业设计的爬虫，爬取58同城、赶集网、链家、安居客、我爱我家网站的房价交易数据。

  Language:Python318121

• widy28/scrapy-taobao

  scrapy模拟淘宝登陆

  Language:Python7436

• geekan/scrapy-examples

  Multifarious Scrapy examples. Spiders for alexa / amazon / douban / douyu / github / linkedin etc.

  Language:Python3.2k1k

• wuchong/scrapy-dynamic-configurable

  A dynamic configurable news crawler based Scrapy

  Language:Python16475

• gnemoug/distribute_crawler

  使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现

  Language:Python3.2k1.6k

• boss-mao/scrapy_enterprise_architecture

  python scrapy 企业级分布式爬虫开发架构模板

  Language:Python9159

• scrapy-plugins/scrapy-splash

  Scrapy+Splash for JavaScript integration

  Language:Python3.1k451

• ictar/python-doc

  translate python documents to Chinese for convenient reference 简而言之，这里用来存放那些Python文档君们，并且尽力将其翻译成中文~~

  1.9k665