jtpio/jupyterlab-system-monitor

After installing this plugin, if there are many kernels running, it will become laggy

icankeep opened this issue · 7 comments

icankeep commented

reproduce step

  1. install
conda create -n jupyterlab3.1.18 python=3.6.7  
source activate jupyterlab3.1.18
pip install jupyterlab==3.1.18 jupyterlab-system-monitor

jupyter lab
  1. open many kernels

image

  1. UI interface will become stuck, directory loading and terminal operations will also become very slow
icankeep commented

main thread dump:

sudo py-spy dump -p 69783
Process 69783: /Users/passer/conda/envs/jupyterlab3.1.18/bin/python /Users/passer/conda/envs/jupyterlab3.1.18/bin/jupyter-lab --debug
Python v3.6.7 (/Users/passer/conda/envs/jupyterlab3.1.18/bin/python3.6)

Thread 0x1198A5DC0 (idle): "MainThread"
    cpu_percent (psutil/__init__.py:995)
    get_process_metric_value (jupyter_resource_usage/metrics.py:19)
    <lambda> (jupyter_resource_usage/metrics.py:37)
    <listcomp> (jupyter_resource_usage/metrics.py:40)
    process_metric (jupyter_resource_usage/metrics.py:40)
    get_metric_values (jupyter_resource_usage/metrics.py:62)
    metrics (jupyter_resource_usage/metrics.py:67)
    cpu_metrics (jupyter_resource_usage/metrics.py:82)
    __call__ (jupyter_resource_usage/prometheus.py:33)
    _step (asyncio/tasks.py:180)
    step (nest_asyncio.py:183)
    _run (asyncio/events.py:145)
    _run_once (nest_asyncio.py:111)
    run_until_complete (nest_asyncio.py:75)
    wrapped (jupyter_client/utils.py:23)
    _run (tornado/ioloop.py:905)
    _run_callback (tornado/ioloop.py:741)
    _run (asyncio/events.py:145)
    _run_once (nest_asyncio.py:111)
    run_until_complete (nest_asyncio.py:75)
    wrapped (jupyter_client/utils.py:23)
    _run (tornado/ioloop.py:905)
    _run_callback (tornado/ioloop.py:741)
    _run (asyncio/events.py:145)
    _run_once (nest_asyncio.py:111)
    run_until_complete (nest_asyncio.py:75)
    wrapped (jupyter_client/utils.py:23)
    _run (tornado/ioloop.py:905)
    _run_callback (tornado/ioloop.py:741)
    _run (asyncio/events.py:145)
    _run_once (nest_asyncio.py:111)
    run_until_complete (nest_asyncio.py:75)
    wrapped (jupyter_client/utils.py:23)
    _run (tornado/ioloop.py:905)
    _run_callback (tornado/ioloop.py:741)
    _run (asyncio/events.py:145)
    _run_once (nest_asyncio.py:111)
    run_until_complete (nest_asyncio.py:75)
    wrapped (jupyter_client/utils.py:23)
    _run (tornado/ioloop.py:905)
    _run_callback (tornado/ioloop.py:741)
    _run (asyncio/events.py:145)
    _run_once (nest_asyncio.py:111)
    run_forever (asyncio/base_events.py:427)
    start (tornado/platform/asyncio.py:199)
    start_ioloop (jupyter_server/serverapp.py:2655)
    start (jupyter_server/serverapp.py:2669)
    launch_instance (jupyter_server/extension/application.py:577)
    <module> (jupyter-lab:8)
icankeep commented

$ cat ~/.jupyter/jupyter_server_config.py
import psutil
c.ResourceUseDisplay.track_cpu_percent = True
c.ResourceUseDisplay.mem_limit = psutil.virtual_memory().total
c.ResourceUseDisplay.cpu_limit = psutil.cpu_count()

dleen commented

@icankeep I am having the same issue that you describe! It seems to be related to the configuration:

c.ResourceUseDisplay.track_cpu_percent = True

When I comment out this configuration everything is fine. Enabling it and launching a few kernels causes the UI to become very laggy.

dleen commented

Here's how I easily reproduce this issue. Create multiple notebooks with contents:

import time

i = 0
while True:
    print(f"i={i}")
    i += 1
    time.sleep(1)

Run 4+ kernels all executing this cell.

Open a terminal and (hopefully your key repeat speed is high enough) hold down a character e.g. "x" to get continuous input into the terminal. This should be very smooth, you should see characters appearing rapidly and without pause.

Now relaunch the server with --ResourceUseDisplay.track_cpu_percent=True.

Repeat the process. While holding down a key in the terminal you will notice frequent lags and pauses.

I think this is because the main thread is getting blocked by psutil. I am currently investigating this.

dleen commented

I believe it is due to this line: https://github.com/jupyter-server/jupyter-resource-usage/blob/master/jupyter_resource_usage/server_extension.py#L22

Probably best to take this issue over to that repo.

jtpio commented

Closing as jupyterlab-system-monitor has now been integrated in jupyter-resource-usage: jupyter-server/jupyter-resource-usage#191

Feel free to open a new issue on the jupyter-resource-usage repo if needed: https://github.com/jupyter-server/jupyter-resource-usage

Thanks!