canonical/notebook-operators

jupyter ui stalls and becomes unusable

Opened this issue · 4 comments

nishant-dash commented

Bug Description

Jupyter ui just starts stalling and I can even stop a notebook. I get either internal errors in the ui or

0: Http failure response for https://<REDACTED>/jupyter/api/namespaces/admin/notebooks/test-dash-2: 0 Unknown Error

After recreating the jupyter ui pod, things work.

To Reproduce

N/A

Environment

intermediate version of kf 1.7
intermediate version of jupyter ui

Relevant Log Output

In the jupyter ui container, I see tracebacks of 

2024-02-21T15:44:27.541Z [jupyter-ui] 2024-02-21 15:44:27,541 | apps.common.utils | INFO | Using config file: /etc/config/spawner_ui_config.yaml
2024-02-21T15:44:27.563Z [jupyter-ui] 2024-02-21 15:44:27,563 | apps.common.utils | INFO | Using config file: /etc/config/spawner_ui_config.yaml
2024-02-21T15:44:27.586Z [jupyter-ui] 2024-02-21 15:44:27,586 | apps.common.utils | INFO | Using config file: /etc/config/spawner_ui_config.yaml
2024-02-21T15:44:27.609Z [jupyter-ui] 2024-02-21 15:44:27,609 | apps.common.utils | INFO | Using config file: /etc/config/spawner_ui_config.yaml
2024-02-21T15:44:27.609Z [jupyter-ui] 2024-02-21 15:44:27,609 | kubeflow.kubeflow.crud_backend.errors.handlers | ERROR | Caught and unhandled Exception!
2024-02-21T15:44:27.610Z [jupyter-ui] 2024-02-21 15:44:27,609 | kubeflow.kubeflow.crud_backend.errors.handlers | ERROR | 'image'
2024-02-21T15:44:27.610Z [jupyter-ui] Traceback (most recent call last):
2024-02-21T15:44:27.610Z [jupyter-ui]   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1820, in full_dispatch_request
2024-02-21T15:44:27.610Z [jupyter-ui]     rv = self.dispatch_request()
2024-02-21T15:44:27.610Z [jupyter-ui]   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1796, in dispatch_request
2024-02-21T15:44:27.610Z [jupyter-ui]     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
2024-02-21T15:44:27.610Z [jupyter-ui]   File "/src/apps/common/routes/get.py", line 47, in get_notebooks
2024-02-21T15:44:27.610Z [jupyter-ui]     contents = [utils.notebook_dict_from_k8s_obj(nb) for nb in notebooks]
2024-02-21T15:44:27.610Z [jupyter-ui]   File "/src/apps/common/routes/get.py", line 47, in <listcomp>
2024-02-21T15:44:27.610Z [jupyter-ui]     contents = [utils.notebook_dict_from_k8s_obj(nb) for nb in notebooks]
2024-02-21T15:44:27.610Z [jupyter-ui]   File "/src/apps/common/utils.py", line 135, in notebook_dict_from_k8s_obj
2024-02-21T15:44:27.610Z [jupyter-ui]     "image": cntr["image"],
2024-02-21T15:44:27.610Z [jupyter-ui] KeyError: 'image'

Additional Context

Thanks @pedrofragola for pointing to kubeflow/kubeflow#6886

syncronize-issues-to-jira commented

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5362.

This message was autogenerated

nishant-dash commented

maybe the python packages are doing something different now? and this spawner ui config yaml is incompatible?
(given that this is an older version of jupyter ui)

pedrofragola commented

plz check the issue kubeflow/kubeflow#6886

nishant-dash commented

Hi given what @pedrofragola linked, this seems like a serious issue since any user can easily or inadvertently DOS the entire jupyter ui service