/jupyterhub-idle-culler

JupyterHub service to cull idle servers and users

Primary LanguagePythonOtherNOASSERTION

JupyterHub Idle Culler Service

GitHub Workflow Status - Test Latest PyPI version GitHub Discourse Gitter

jupyterhub-idle-culler provides a JupyterHub service to identify and shut down idle or long-running Jupyter Notebook servers. The exact actions performed are dependent on the used spawner for the Jupyter Notebook server (e.g. the default LocalProcessSpawner, kubespawner, or dockerspawner). In addition, if explicitly requested, all users whose Jupyter Notebook servers have been shut down this way are deleted as JupyterHub users from the internal database. This neither affects the authentication method which continues to allow those users to log in nor does it delete persisted user data (e.g. stored in docker volumes for dockerspawner or in persisted volumes for kubespawner).

Setup

Installation

pip install jupyterhub-idle-culler

Permissions

Prior to JupyterHub 2.0, the jupyterhub-idle-culler required full administrative privileges, in order to have sufficient permissions to stop servers on behalf of users.

JupyterHub 2.0 introduces scopes to allow for more fine-grained permission control. This means that the configured culler service does not need full administrative privileges anymore. It can be assigned only the permissions it needs.

jupyterhub-idle-culler requires the following scopes to function:

  • list:users - to access to the user list API, our source of information about who to cull
  • read:users:activity - to read the users' last_activity field
  • read:servers - to read the users' servers field
  • delete:servers - to stop users' servers, and delete named servers if --remove-named-servers is passed
  • admin:users (optional) - to delete users if --cull-users is passed

To assign the service the appropriate permissions, declare a role in your jupyterhub_config.py:

c.JupyterHub.load_roles = [
    {
        "name": "jupyterhub-idle-culler-role",
        "scopes": [
            "list:users",
            "read:users:activity",
            "read:servers",
            "delete:servers",
            # "admin:users", # if using --cull-users
        ],
        # assignment of role's permissions to:
        "services": ["jupyterhub-idle-culler-service"],
    }
]

As a hub managed service

In jupyterhub_config.py, add the following dictionary for the idle-culler Service to the c.JupyterHub.services list:

c.JupyterHub.services = [
    {
        "name": "jupyterhub-idle-culler-service",
        "command": [
            sys.executable,
            "-m", "jupyterhub_idle_culler",
            "--timeout=3600",
        ],
        # "admin": True,
    }
]

where:

  • "command" indicates that the Service will be managed by the Hub, and
  • "admin": True grants admin permissions to this Service and is only meant for use with jupyterhub < 2.0; see [above][permissions].

As a standalone script

jupyterhub-idle-culler can also be run as a standalone script. It can access the hub's api with a service token.

Register the service token with JupyterHub in jupyterhub_config.py:

c.JupyterHub.services = [
    {
        "name": "jupyterhub-idle-culler-service",
        "api_token": "...",
        # "admin": True,
    }
]

where:

  • "api_token" contains a secret token, e.g. generated by openssl rand -hex 32, and
  • "admin": True grants admin permissions to this Service and is only meant for use with jupyterhub < 2.0; see [above][permissions].

and store the same token in a JUPYTERHUB_API_TOKEN environment variable. Then start jupyterhub-idle-culler manually.

export JUPYTERHUB_API_TOKEN=api_token_above...
python3 -m jupyterhub-idle-culler [--timeout=900] [--url=http://localhost:8081/hub/api]

Command line flags

  --api-page-size                  Number of users to request per page, when
                                   using JupyterHub 2.0's paginated user list
                                   API. Default: user the server-side default
                                   configured page size. (default 0)
  --concurrency                    Limit the number of concurrent requests made
                                   to the Hub.  Deleting a lot of users at the
                                   same time can slow down the Hub, so limit
                                   the number of API requests we have
                                   outstanding at any given time. (default 10)
  --cull-admin-users               Whether admin users should be culled (only
                                   if --cull-users=true). (default True)
  --cull-every                     The interval (in seconds) for checking for
                                   idle servers to cull. (default 0)
  --cull-users                     Cull users in addition to servers.  This is
                                   for use in temporary-user cases such as
                                   tmpnb. (default False)
  --internal-certs-location        The location of generated internal-ssl
                                   certificates (only needed with --ssl-
                                   enabled=true). (default internal-ssl)
  --max-age                        The maximum age (in seconds) of servers that
                                   should be culled even if they are active.
                                   (default 0)
  --remove-named-servers           Remove named servers in addition to stopping
                                   them.  This is useful for a BinderHub that
                                   uses authentication and named servers.
                                   (default False)
  --ssl-enabled                    Whether the Jupyter API endpoint has TLS
                                   enabled. (default False)
  --timeout                        The idle timeout (in seconds). (default 600)
  --url                            The JupyterHub API URL.

Caveats

  1. last_activity is not updated with high frequency, so cull timeout should be greater than the sum of:

    • single-user websocket ping interval (default: 30 seconds)
    • JupyterHub.last_activity_interval (default: 5 minutes)

  2. The same --timeout and --max-age values are used to cull users and users' servers. If you want a different value for users and servers, you should add this script to the services list twice, just with different names, different values, and one with the --cull-users option.

  3. By default HTTP requests to the hub timeout after 60 seconds. This can be changed by setting the JUPYTERHUB_REQUEST_TIMEOUT environment variable.

How it works

jupytehrub-idle-culler lists available users via JupyterHub's /users REST API.

jupyterhub-idle-culler culls user servers using JupyterHub's REST API (/users/{name}/server or /users/{name}/servers/{server_name}), and makes the culling decisions based on its configuration and what JupyterHub reports about the user servers via its REST API (/users) where user servers' last_activity is reported back.

The last_activity that JupyterHub reports is the most recent summary of information updated at a regular interval via the update_last_activity function that combines two sources of information.

  1. The proxy's routes data

    The update_last_activity function will ask the proxy for the active routes like /user/user1 and collects associated last_activity data if it is available. This activity represents successfully proxies network traffic.

    last_activity data for routes will be available when using configurable-http-proxy as JupyterHub does by default, but if for example traefik-proxy is used as it is in the TLJH distribution, no such data will be available.

  2. The user server's activity reports

    The update_last_activity function also reads JupyterHub's database that keeps state about servers last_activity. These database records are updated whenever a server notifies JupyterHub about activity, as they are responsible to do.

    Servers notify JupyterHub about activity by being started by the jupyterhub-singleuser script that is made available by installing jupyterhub (or jupyterhub-base on conda-forge).

    The jupyterhub-singleuser script launches a modified server application that keeps JupyterHub updated with the server activity via the notify_activity function.

    The notify_activity function in turn make use of the server applications last_activity function (see implementation in NotebookApp and ServerApp respectively) that that combines information from API activity, kernel activity, kernel shutdown, and terminal activity. This activity also covers activity of applications like RStudio running via jupyter-server-proxy.

Here is a summary of what's described so far:

  1. jupyterhub-idle-culler culls servers via JupyterHub's REST API.
  2. jupyterhub-idle-culler makes decisions based on information retrieved by JupyterHub REST API.
  3. JupyterHub REST API reports information regularly updated by summarizing information gained by: asking the proxy about routes' activity, and by retaining activity information reported by the servers.

Now, as the server's kernel activity influence the activity that servers will notify JupyterHub about, the kernel activity in turn influences jupyterhub-idle-culler. Due to this, it can be relevant to also learn a little about a mechanism to cull idle kernels as well even though jupyterhub-idle-culler isn't involved in that.

The default kernel manager, the MappingKernelManager, can be configured to cull idle kernels. Its configuration is documented in NotebookApp's and ServerApp's respective documentation, and here are some relevant kernel culling configuration options:

  • MappingKernelManager.cull_busy

  • MappingKernelManager.cull_idle_timeout

  • MappingKernelManager.cull_interval

  • MappingKernelManager.cull_connected

    Note that cull_connected can be tricky to understand for JupyterLab as a browser having a web-socket connection to a kernel or not isn't as obvious as it was in the classical Jupyter notebook UI. See this issue for more details.

    Also note that configuration of MappingKernelManager should be made on the user server itself, for example via a jupyter_notebook_config.py file in /etc/jupyter or /usr/local/etc/jupyter rather than where JupyterHub is running.

Finally, note that a Jupyter Notebook server can shut itself down without without intervention by jupyterhub-idle-culler if NotebookApp.shutdown_no_activity_timeout is configured.

Caveats

Pagination

JupyterHub 2.0 introduces pagination to the /users API endpoint. This pagination does not guarantee a consistent snapshot for consecutive requests spread over time, so it is possible for a highly active hub to occasionally miss culling users crossing page boundaries between requests. This is expected to be an infrequent occurrence and only result in delaying a server being culled by one cull interval in realistic scenarios, so of minor consequence in JupyterHub.

The issue can be mitigated by requesting a larger page size, via e.g. --api-page-size=200, but feel free to open an issue if this is causing a problem for you.