Thriftpy/gunicorn_thrift

The gunicorn got timeout but not exit

Closed this issue · 1 comments

I'm use the gunicorn_thrift like last issues,some time is run well ,but when it run a while ,I got a error

1741648 [2017-12-06 12:15:39 +0000] [5727] [CRITICAL] WORKER TIMEOUT (pid:7438)
1741649 [2017-12-06 12:15:39 +0000] [5727] [CRITICAL] WORKER TIMEOUT (pid:7445)
1741650 [2017-12-06 12:15:39 +0000] [5727] [CRITICAL] WORKER TIMEOUT (pid:7447)
1741651 [2017-12-06 12:15:39 +0000] [5727] [CRITICAL] WORKER TIMEOUT (pid:7448)
1741652 [2017-12-06 12:15:39 +0000] [5727] [CRITICAL] WORKER TIMEOUT (pid:7449)
1741653 [2017-12-06 12:15:39 +0000] [5727] [CRITICAL] WORKER TIMEOUT (pid:7451)
1741654 [2017-12-06 12:15:39 +0000] [5727] [CRITICAL] WORKER TIMEOUT (pid:7453)
1741655 [2017-12-06 12:15:39 +0000] [5727] [CRITICAL] WORKER TIMEOUT (pid:7456)
1741656 [2017-12-06 12:15:39 +0000] [5727] [CRITICAL] WORKER TIMEOUT (pid:7457)
1741657 [2017-12-06 12:15:39 +0000] [5727] [CRITICAL] WORKER TIMEOUT (pid:7458)
1741658 [2017-12-06 12:15:39 +0000] [5727] [CRITICAL] WORKER TIMEOUT (pid:7459)
1741659 [2017-12-06 12:15:40 +0000] [5727] [CRITICAL] WORKER TIMEOUT (pid:7434)
1741660 [2017-12-06 12:15:40 +0000] [5727] [CRITICAL] WORKER TIMEOUT (pid:7437)
1741661 [2017-12-06 12:15:40 +0000] [5727] [CRITICAL] WORKER TIMEOUT (pid:7443)
1741662 [2017-12-06 12:15:40 +0000] [7506] [INFO] Booting worker with pid: 7506
1741663 [2017-12-06 12:15:40 +0000] [7507] [INFO] Booting worker with pid: 7507
1741664 [2017-12-06 12:15:40 +0000] [7508] [INFO] Booting worker with pid: 7508
1741665 [2017-12-06 12:15:40 +0000] [7509] [INFO] Booting worker with pid: 7509
1741666 [2017-12-06 12:15:40 +0000] [7510] [INFO] Booting worker with pid: 7510
1741667 [2017-12-06 12:15:40 +0000] [7511] [INFO] Booting worker with pid: 7511
1741668 [2017-12-06 12:15:40 +0000] [7512] [INFO] Booting worker with pid: 7512
1741669 [2017-12-06 12:15:40 +0000] [7513] [INFO] Booting worker with pid: 7513
1741670 [2017-12-06 12:15:40 +0000] [7514] [INFO] Booting worker with pid: 7514
1741671 [2017-12-06 12:15:40 +0000] [7515] [INFO] Booting worker with pid: 7515
1741672 [2017-12-06 12:15:40 +0000] [7514] [ERROR] Exception in worker process:
1741673 Traceback (most recent call last):
1741674   File "/usr/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 507, in spawn_worker
1741675     worker.init_process()
1741676   File "/usr/local/lib/python2.7/site-packages/gunicorn_thrift/thriftpy_gevent_worker.py", line 98, in init_process
1741677     return super(GeventThriftPyWorker, self).init_process()
1741678   File "/usr/local/lib/python2.7/site-packages/gunicorn/workers/ggevent.py", line 192, in init_process
1741679     super(GeventWorker, self).init_process()
1741680   File "/usr/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 118, in init_process
1741681     self.wsgi = self.app.wsgi()
1741682   File "/usr/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
1741683     self.callable = self.load()
1741684   File "/usr/local/lib/python2.7/site-packages/gunicorn_thrift/thriftapp.py", line 39, in load
1741685     self.thrift_app = self.load_thrift_app()
1741686   File "/usr/local/lib/python2.7/site-packages/gunicorn_thrift/thriftapp.py", line 33, in load_thrift_app
1741687     return utils.load_obj(self.app_uri)
1741688   File "/usr/local/lib/python2.7/site-packages/gunicorn_thrift/utils.py", line 16, in load_obj
1741689     mod = importlib.import_module(module)
1741690   File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
1741691     __import__(name)
1741692   File "/usr/local/lib/python2.7/site-packages/gevent-1.1rc2-py2.7-linux-x86_64.egg/gevent/builtins.py", line 58, in __import__
1741693     result = _import(*args, **kwargs)
1741694   File "/home/zhouyi/heybox_script/release/pubg_thriftpy/pubg_account_server.py", line 13, in <module>
1741695     pg_account = psycopg2.connect(database="account", user='postgres', password="", host="10.10.10.54", port='19934')
1741696   File "/usr/local/lib/python2.7/site-packages/psycopg2/__init__.py", line 164, in connect
1741697     conn = _connect(dsn, connection_factory=connection_factory, async=async)
1741698 OperationalError: FATAL:  sorry, too many clients already

it seems like the WORKER got timeout but not exit so the master booting another worker , finally the worker connect to pg got the ' too many clients' error, anyone know what's the problem ?

There are more detail #1657