TeamHG-Memex/scrapy-rotating-proxies

raise error.ReactorNotRestartable() twisted.internet.error.ReactorNotRestartable

Arregator opened this issue · 1 comments

Hi,

when I uncomment rotating_proxies.middlewares in my settings.py I get error:

...
2020-03-10 17:59:55 [scrapy.core.engine] INFO: Spider opened
2020-03-10 17:59:55 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-03-10 17:59:55 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6024
2020-03-10 17:59:55 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 0, unchecked: 20, reanimated: 0, mean backoff time: 0s)
Traceback (most recent call last):
  File "c:\Users\<user>\.vscode\extensions\ms-python.python-2020.2.64397\pythonFiles\ptvsd_launcher.py", line 48, in <module>
    main(ptvsdArgs)
  File "c:\Users\<user>\.vscode\extensions\ms-python.python-2020.2.64397\pythonFiles\lib\python\old_ptvsd\ptvsd\__main__.py", line 432, in main
    run()
  File "c:\Users\<user>\.vscode\extensions\ms-python.python-2020.2.64397\pythonFiles\lib\python\old_ptvsd\ptvsd\__main__.py", line 316, in run_file
    runpy.run_path(target, run_name='__main__')
  File "C:\Users\<user>\Anaconda3\envs\scrap\lib\runpy.py", line 263, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "C:\Users\<user>\Anaconda3\envs\scrap\lib\runpy.py", line 96, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "C:\Users\<user>\Anaconda3\envs\scrap\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "c:\Users\<user>\Documents\SkillFactory\carsSpider\carsSpider\spiders\carsSpider_spider.py", line 164, in <module>  
    process.start() # the script will block here until the crawling is finished
  File "C:\Users\<user>\Anaconda3\envs\scrap\lib\site-packages\scrapy\crawler.py", line 293, in start
    reactor.run(installSignalHandlers=False)  # blocking call
  File "C:\Users\<user>\Anaconda3\envs\scrap\lib\site-packages\twisted\internet\base.py", line 1282, in run
    self.startRunning(installSignalHandlers=installSignalHandlers)
  File "C:\Users\<user>\Anaconda3\envs\scrap\lib\site-packages\twisted\internet\base.py", line 1262, in startRunning       
    ReactorBase.startRunning(self)
  File "C:\Users\<user>\Anaconda3\envs\scrap\lib\site-packages\twisted\internet\base.py", line 765, in startRunning        
    raise error.ReactorNotRestartable()
twisted.internet.error.ReactorNotRestartable

this is my settintg.py

DOWNLOADER_MIDDLEWARES = {
    'rotating_proxies.middlewares.RotatingProxyMiddleware': 610,
    'rotating_proxies.middlewares.BanDetectionMiddleware': 620,
}

ROTATING_PROXY_LIST = [

    '<user>@<password>:<ip1>:<port>',
    '<user>@<password>:<ip2>:<port>',
    '<user>@<password>:<ip3>:<port>',

]

If I disbale rotating_proxies.middlewares - it runs ok

Can you please help me with tackling this issue ?

Thank you,
Roman.

The issues seems was caused not by this software.

Basically, the error was because the code running a spider as a script via CrawlerProcess object was pasted into the same file as defined spider class and was called each time scrapy imported class definition.