Python3WebSpider/ProxyPool

dev环境启动scheduler时server模块报错

yjshi2015 opened this issue · 2 comments

yjshi2015 commented

Describe the bug
本地dev环境启动scheduler时，server模块启动报错，提示：在一个非套接字上尝试了一个操作，无法提供HTTP服务。如果只是运行server类，则可以。

To Reproduce
直接运行scheduler的main方法。

Screenshots
image

报错信息如下：
`
2022-06-22 19:38:44.626 | INFO | main:run:90 - starting tester, pid None...
2022-06-22 19:38:44.641 | INFO | main:run:95 - starting getter, pid None....
2022-06-22 19:38:44.651 | INFO | main:run:100 - starting server, pid None...
2022-06-22 19:38:46.473 | DEBUG | mp_main:run_tester:25 - tester loop 0 start ....
2022-06-22 19:38:46.474 | INFO | proxypool.processors.tester:run:59 - starting tester......
2022-06-22 19:38:46.475 | DEBUG | proxypool.processors.tester:run:61 - 7312 proxies to test
2022-06-22 19:38:46.476 | DEBUG | proxypool.processors.tester:run:64 - testing proxies use cursor 0, count 20

  • Serving Flask app 'proxypool.processors.server' (lazy loading)
  • Environment: production
    WARNING: This is a development server. Do not use it in a production deployment.
    Use a production WSGI server instead.
  • Debug mode: on
  • Running on all addresses (0.0.0.0)
    WARNING: This is a development server. Do not use it in a production deployment.
  • Running on http://127.0.0.1:5555
  • Running on http://192.168.xxx.xxx:5555 (Press CTRL+C to quit)
    2022-06-22 19:38:46.527 | DEBUG | proxypool.processors.tester:test:31 - testing 47.254.92.194:7328
  • Restarting with stat
    2022-06-22 19:38:46.533 | DEBUG | proxypool.processors.tester:test:31 - testing 171.92.21.162:9000
    2022-06-22 19:38:46.533 | DEBUG | proxypool.processors.tester:test:31 - testing 103.137.91.250:8080
    2022-06-22 19:38:50.619 | DEBUG | proxypool.processors.tester:test:31 - testing 171.92.21.162:9000
  • Debugger is active!
  • Debugger PIN: 127-838-931
    Process Process-3:
    Traceback (most recent call last):
    File "D:\Program Files (x86)\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
    File "D:\Program Files (x86)\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    File "D:\idea_workspace\SpiderProxyPool\proxypool\scheduler.py", line 81, in run_server
    app.run(host=API_HOST, port=API_PORT, threaded=API_THREADED)
    File "D:\Program Files (x86)\lib\site-packages\flask\app.py", line 920, in run
    run_simple(t.cast(str, host), port, self, **options)
    File "D:\Program Files (x86)\lib\site-packages\werkzeug\serving.py", line 1065, in run_simple
    srv = make_server(
    File "D:\Program Files (x86)\lib\site-packages\werkzeug\serving.py", line 849, in make_server
    return ThreadedWSGIServer(
    File "D:\Program Files (x86)\lib\site-packages\werkzeug\serving.py", line 716, in init
    self.socket = socket.fromfd(fd, address_family, socket.SOCK_STREAM)
    File "D:\Program Files (x86)\lib\socket.py", line 545, in fromfd
    nfd = dup(fd)
    OSError: [WinError 10038] 在一个非套接字上尝试了一个操作。
    2022-06-22 19:38:50.740 | DEBUG | mp_main:run_getter:37 - getter loop 0 start....
    2022-06-22 19:38:50.743 | INFO | proxypool.processors.getter:run:25 - crawler <public.Fatezero.FatezeroCrawler object at 0x000002AF5A4CAF20> to get proxy
    2022-06-22 19:38:50.744 | INFO | proxypool.crawlers.base:crawl:20 - fetching http://proxylist.fatezero.org/proxy.list
    2022-06-22 19:38:50.907 | INFO | proxypool.crawlers.base:crawl:23 - fetched proxy 51.81.80.109:3128 from http://proxylist.fatezero.org/proxy.list

`

Environments (please complete the following information):

  • win10
  • Python 3.10
yjshi2015 commented

找到原因了，应该是依赖的模块的版本问题。我是把所有的依赖都安装最高版本，没有参照requirements.txt来安装依赖，所以调整后就跑通了。

Germey commented

多谢，这个问题我没有复现，解决就好。