gnemoug/distribute_crawler
使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现
Python
Issues
- 0
不好意思 ,初学请教一个问题
#33 opened by echoocking - 1
有人用python3的venv成功连上pymongo吗
#36 opened by xiaocwan - 0
请问这个项目是怎么创建的呢(关于graphite部分不懂,scrapy框架是懂的)
#35 opened by WANGYANG12138 - 5
执行scrapy crawl woaidu出错
#13 opened by josephok - 2
scrapy的版本是多少?
#32 opened by Bajie-cigarette - 1
- 0
下载docx文件,打开出错
#31 opened by summerYhl - 0
- 0
- 0
- 4
分布式是如何体现的?
#25 opened by lywhlao - 5
运行scrapy crawler woaidu之后,卡住不动了
#14 opened by MRLuowen - 2
socket.error: [Errno 10061]
#21 opened by ichenfujun - 4
多个pipelines如何指定的?
#7 opened by xiaonan - 1
- 0
有用过phantomjs的server模块?
#22 opened by rfyiamcool - 0
hi,好奇对于js的抓取你是如何实现的
#20 opened by rfyiamcool - 1
怎么判断URL是不是该抓取
#18 opened by rfyiamcool - 0
有相关的性能报告么?
#19 opened by rfyiamcool - 0
关于js渲染的是如何处理的呢?
#17 opened by rfyiamcool - 1
是否把readme.md 翻译成英文更好一点,或者提供一个英文版本?
#16 opened by viciousstar - 3
ImportError: Error loading object 'woaidu_crawler.pipelines.bookfile.WoaiduBookFile': cannot import name MediaPipeline
#8 opened by lifehack - 0
- 1
process_item
#12 opened by pazingaa - 0
redis-py
#11 opened by qichuan8953 - 2
scrapy的文档中没有scheduler的api,作者是如何实现重写的?
#10 opened by yanpeipan - 0
- 1
- 1
BUG in graphite.py
#3 opened by sailaway - 1
can't run the cralwer
#2 opened by yuany