Some DB issue that crashes the app.
Opened this issue · 2 comments
Sometimes just logging in would crash the app.
This is what I find in the logs:
172.16.140.84 - - [10/May/2022:12:01:31 +0000] "GET / HTTP/1.1" 200 3098 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) coc_coc_browser/80.0.180 Chrome/74.0.3729.180 Safari/537.36"
[2022-05-10 12:42:50,629] ERROR in app: Exception on /token/refresh [POST]
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1820, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functionsrule.endpoint
File "/usr/local/lib/python3.6/site-packages/flask_restful/init.py", line 467, in wrapper
resp = resource(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask/views.py", line 89, in view
return self.dispatch_request(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask_restful/init.py", line 582, in dispatch_request
resp = meth(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask_jwt_extended/view_decorators.py", line 160, in wrapper
return fn(*args, **kwargs)
File "/home/servicex/servicex/resources/users/token_refresh.py", line 38, in post
user = UserModel.find_by_sub(get_jwt_identity())
File "/home/servicex/servicex/models.py", line 74, in find_by_sub
return cls.query.filter_by(sub=sub).first()
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2819, in first
return self.limit(1)._iter().first()
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2906, in _iter
execution_options={"_sa_orm_load_options": self.load_options},
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1692, in execute
result = conn._execute_20(statement, params or {}, execution_options)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 326, in _execute_on_connection
self, multiparams, params, execution_options
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1508, in _execute_clauseelement
cache_hit=cache_hit,
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1863, in execute_context
e, statement, parameters, cursor, context
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2044, in handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from=e
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 207, in raise
raise exception
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1820, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
[SQL: SELECT users.admin AS users_admin, users.created_at AS users_created_at, users.email AS users_email, users.experiment AS users_experiment, users.id AS users_id, users.institution AS users_institution, users.name AS users_name, users.pending AS users_pending, users.refresh_token AS users_refresh_token, users.sub AS users_sub, users.updated_at AS users_updated_at
FROM users
WHERE users.sub = %(sub_1)s
LIMIT %(param_1)s]
[parameters: {'sub_1': 'a51dbd7e-d274-11e5-9b11-9be347a09ce0', 'param_1': 1}]
(Background on this error at: https://sqlalche.me/e/14/e3q8)
@ivukotic can you see if the Postgres pod is ok? That would seem like the most likely issue with this error.
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
Seems postgres is OK. It did have 10 restarts but all before 23rd May.
We should increase request on its memory. We request 256MB and currently using 442MB.