Lost config from Zookeeper makes spider down

[jamesliu668]

Hello,

I have deployed my scrapy-cluster on cloud platform with 3 kafka/zookeeper, 1 redis, and 1 spider machine. All hosts can communicate with each other in the local network. The spider machine can access the internet. Currently I am suffering from this error "Lost config from Zookeeper". But I double check my kafka/zookeeper hosts, all are online. After I kill the spider and restart it, it can work properly. But after several days, it will get this error and stop again.

Here is the full error msg:
`
2018-10-06 23:09:35,132 [sc-crawler] INFO: Lost config from Zookeeper

Exception in thread Thread-4:

Traceback (most recent call last):

File "/usr/local/python2.7/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()

File "/usr/local/python2.7/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)

File "/usr/local/python2.7/lib/python2.7/site-packages/kazoo/protocol/connection.py", line 471, in zk_loop
if retry(self._connect_loop, retry) is STOP_CONNECTING:

File "/usr/local/python2.7/lib/python2.7/site-packages/kazoo/retry.py", line 123, in call
return func(*args, **kwargs)

File "/usr/local/python2.7/lib/python2.7/site-packages/kazoo/protocol/connection.py", line 488, in _connect_loop
status = self._connect_attempt(host, port, retry)

File "/usr/local/python2.7/lib/python2.7/site-packages/kazoo/protocol/connection.py", line 529, in _connect_attempt
[], [], timeout)[0]

File "/usr/local/python2.7/lib/python2.7/site-packages/kazoo/handlers/threading.py", line 147, in select
return select.select(*args, **kwargs)

TypeError: argument must be an int, or have a fileno() method.`

I am under python 2.7.

Thanks

[madisonb]

Given that the traceback looks like it is coming from within Kazoo, I am not sure I can help you debug unless you can give me steps to reproduce it within the project. If you can guide me in generating this error with the docker containers or on the vagrant machine, that would be helpful.

Otherwise, given how dormant this issue has been, I am going to close it

[jamesliu668]

It's a little bit weird as when I start the spider with "scrapy runspider xx.py". It's working properly at the beginning. But somehow this error message will appear after several days running.

[madisonb]

Closing due to inactivity and no steps to reproduce