istresearch/scrapy-cluster

Issues

• runspider: error: Unable to load 'link_spider.py': attempted relative import with no known parent package

  #268 opened a year ago by BeamoINT
  4

• ERROR: Unable to connect to Kafka in Pipeline due to attempt to connect already-connected SSLSocket!, raising exit flag.

  #269 opened a year ago by BeamoINT
  1

• Is this project no longer supported?

  #266 opened a year ago by flyingdev
  2

• ui exception No connection adapters were found

  #260 opened 3 years ago by new-wxw
  0

• Crawler unhandled exceptions not logged

  #259 opened 3 years ago by getorca
  1

• scrapy-cluster don't scroll the entire pages

  #258 opened 3 years ago by RochdiBoudokhane
  7

• Crawl complete signal

  #193 opened 6 years ago by wysiwygism
  5

• Stats keys flooding redis + Crawl resilience + retry strategy

  #255 opened 4 years ago by benjaminelkrieff
  5

• Jobs disappearing: is there a way to monitor crawls ?

  #251 opened 4 years ago by benjaminelkrieff
  4

• Incremented fail stats

  #254 opened 4 years ago by mingxuan1
  0

• Upgrading the ELK stack

  #253 opened 4 years ago by 4OH4
  2

• Scrape delay

  #250 opened 4 years ago by tluyben
  2

• TypeError: can't pickle thread.lock objects

  #252 opened 4 years ago by benjaminelkrieff
  2

• Future of the project

  #235 opened 4 years ago by demisx
  9

• maxdepth can not large than 2

  #241 opened 4 years ago by anthony9981
  5

• Python 3

  #230 opened 5 years ago by deed02392
  6

• Update Releases, Docker compose Issues

  #238 opened 4 years ago by MrMoon
  2

• Any suggestion on the best way to get the crawled data out to a file?

  #232 opened 5 years ago by ahoquegh
  0

• Docker ERROR: Could not ping Zookeeper

  #220 opened 5 years ago by Shique
  6

• How to submit a POST request through crawler REST API

  #231 opened 5 years ago by YanzhongSu
  2

• A mistake in parameter deliver through two method!

  #225 opened 5 years ago by kevin-ZZZ
  4

• supervisord install error while vagrant up

  #226 opened 5 years ago by fatemeeeeeee
  2

• Why encoding byted body

  #229 opened 5 years ago by YanzhongSu
  2

• online integration test error

  #224 opened 5 years ago by rawiafray
  2

• Can't up cluster from master branch

  #227 opened 5 years ago by MasterSergius
  1

• `Demo.incoming` topic keeps piling up

  #221 opened 5 years ago by YanzhongSu
  6

• Cannot turn off DEBUG level log

  #223 opened 5 years ago by YanzhongSu
  2

• Plans to support latest version of Kafka, ELK & Scrapy?

  #217 opened 5 years ago by kumaresra
  1

• how to process different type of item in Processor?

  #219 opened 5 years ago by ChengkaiYang2022
  3

• Replace Kafka with another pub/sub service?

  #215 opened 5 years ago by jrmlhermitte
  3

• what's the purpose of item_copy in pipelines.py?

  #218 opened 5 years ago by ChengkaiYang2022
  2

• Lost config from Zookeeper makes spider down

  #203 opened 6 years ago by jamesliu668
  3

• cluster mode online test hangs

  #213 opened 6 years ago by danmsf
  3

• Website did't response for a long time,how to solve the problem

  #214 opened 6 years ago by Johnson0016
  1

• How does it work with CrawlSpider?

  #212 opened 6 years ago by kevin-ZZZ
  1

• Fill a single item from many requests / "can't pickle thread.lock objects"

  #210 opened 6 years ago by llermaly
  3

• Does it support python3?

  #211 opened 6 years ago by xingzhicn
  1

• Docker compose : kafka and crawler containers restarting forever

  #206 opened 6 years ago by llermaly
  6

• Rest - signal only works in main thread

  #204 opened 6 years ago by Shique
  2

• Kafka-Monitor stats can potentially accumulate indefinitely if Redis restarts/fails.

  #186 opened 6 years ago by devspyrosv
  2

• Add login in LinkSpider

  #188 opened 6 years ago by josselinlbe
  2

• scrapy-cluster stops crawling of some websites after a while

  #196 opened 6 years ago by alotfitakami
  2

• Rebuilding crawler container fails

  #197 opened 6 years ago by mrvnklm
  1

• Missing KAFKA_PRODUCER_TOPIC in docker rest settings

  #194 opened 6 years ago by aafeher
  1

• Getting twisted.web._newclient.ResponseNeverReceived exception

  #189 opened 6 years ago by spidysenses
  2

• Docker vs Vagrant

  #192 opened 6 years ago by mshahriarinia
  1

• Help processing pages returned by crawler

  #187 opened 6 years ago by hellsingnorevy
  2

• Support for custom header and cookies for the initial request from kafka_monitor.py feed

  #182 opened 7 years ago by knirbhay
  5

• Typo on document of Kafka-monitor API section

  #183 opened 6 years ago by chihkaiyu
  2

• Inter-spider communication

  #178 opened 7 years ago by shenbakeshkishore
  1