jupyterhub/dockerspawner

[DOC] DockerSpawner timeout with custom image

lidongze0629 opened this issue · 2 comments

lidongze0629 commented

Bug description

Follow doc, I had build a custom image with follow command

# Dockerfile
FROM python:3.6
RUN pip3 install \
    jupyterhub==0.7.2 \
    'notebook>=5.0,<=6.0'

# create a user, since we don't want to run as root
RUN useradd -m jovyan
ENV HOME=/home/jovyan
WORKDIR $HOME
USER jovyan

CMD ["jupyterhub-singleuser"]

# shell command
docker build -t graphscope/notebook:base --network=host -f ./Dockerfile  .
docker push graphscope/notebook:base

Then restart my jupyterhub service, and login with githubOA, the following error message appeared.

[I 2021-01-14 11:11:03.796 JupyterHub dockerspawner:1009] Starting container jupyter-lidongze0629 (id: b8d903c)
[I 2021-01-14 11:11:04.710 JupyterHub log:181] 302 GET /hub/spawn -> /hub/spawn-pending/lidongze0629 (lidongze0629@106.11.34.226) 1003.58ms
[I 2021-01-14 11:11:05.185 JupyterHub pages:402] lidongze0629 is pending spawn
[I 2021-01-14 11:11:05.188 JupyterHub log:181] 200 GET /hub/spawn-pending/lidongze0629 (lidongze0629@106.11.34.226) 4.97ms
[W 2021-01-14 11:11:59.874 JupyterHub user:738] lidongze0629's server never showed up at http://127.0.0.1:32789/user/lidongze0629/ after 60 seconds. Giving up
ERROR:asyncio:Task exception was never retrieved
future: <Task finished name='Task-27' coro=<BaseHandler.spawn_single_user() done, defined at /usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py:791> exception=TimeoutError("Server at http://127.0.0.1:32789/user/lidongze0629/ didn't respond in 60 seconds")>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 991, in spawn_single_user
    await gen.with_timeout(
  File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 902, in finish_user_spawn
    await spawn_future
  File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 719, in spawn
    await self._wait_up(spawner)
  File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 766, in _wait_up
    raise e
  File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 733, in _wait_up
    resp = await server.wait_up(
  File "/usr/local/lib/python3.8/dist-packages/jupyterhub/utils.py", line 235, in wait_for_http_server
    re = await exponential_backoff(
  File "/usr/local/lib/python3.8/dist-packages/jupyterhub/utils.py", line 183, in exponential_backoff
    raise TimeoutError(fail_message)
TimeoutError: Server at http://127.0.0.1:32789/user/lidongze0629/ didn't respond in 60 seconds
[I 2021-01-14 11:11:59.887 JupyterHub log:181] 200 GET /hub/api/users/lidongze0629/server/progress (lidongze0629@106.11.34.226) 54262.15ms

Here is my config.

# jupyterhub_config.py
import json
import os
import sys

from oauthenticator.github import GitHubOAuthenticator

c = get_config()

repo_dir = os.path.join(os.path.dirname(__file__), '../')
with open(os.path.join(repo_dir, 'config', 'config.json')) as f:
    config_json = json.load(f)

c.JupyterHub.ip = '0.0.0.0'

# self-signed ssl cert
c.JupyterHub.ssl_key = os.path.join(repo_dir, 'deploy', 'ssl', 'key.pem')
c.JupyterHub.ssl_cert = os.path.join(repo_dir, 'deploy', 'ssl', 'cert.pem')

# a set of github users
c.Authenticator.admin_users = set(config_json['admin_users'])
c.JupyterHub.admin_access = True  # have permission to log in to the single user notebook servers owned by other users

c.LocalAuthenticator.create_system_users = True  # must be False, we create/remote system user manually

# The docker instances need access to the Hub, so the default loopback port doesn't work:
from jupyter_client.localinterfaces import public_ips
c.JupyterHub.hub_ip = public_ips()[0]

# docker spawner
c.JupyterHub.spawner_class = 'dockerspawner.DockerSpawner'
c.DockerSpawner.image = config_json['singleuser_image']  # graphscope/notebook:base
notebook_dir = '/home/jovyan'  # consistent with the dockerfile
c.DockerSpawner.volumes = { config_json['kubernetes_config_dir']: notebook_dir }
c.DockerSpawner.notebook_dir = notebook_dir  # set workspace
c.DockerSpawner.args = ['--NotebookApp.default_url=/Welcome.ipynb']
c.Spawner.http_timeout = 60  # I also increase to 300, but it doesn't works
c.Spawner.start_timeout = 60  #  I also increase to 300, but it doesn't works
c.JupyterHub.tornado_settings = {'slow_spawn_timeout': 60} # I also increase to 300, but it doesn't works

# github OA
c.JupyterHub.authenticator_class = GitHubOAuthenticator
os.environ['OAUTH_CALLBACK_URL'] = config_json['github_oa']['callback_url']
os.environ['OAUTH_CLIENT_ID'] = config_json['github_oa']['client_id']
os.environ['OAUTH_CLIENT_SECRET'] = config_json['github_oa']['client_secret']

# idle culler
c.JupyterHub.services = [
    {
        'name': 'idle-culler',
        'admin': True,
        'command': [sys.executable, '-m', 'jupyterhub_idle_culler',
                    '--timeout={}'.format(config_json['idle_timeout_seconds']),
                    '--max-age={}'.format(config_json['max_age_timeout_seconds'])
        ],
    }
]

I found that the container exited soon after starting.

docker ps -a
b8d903c43102        graphscope/notebook:base   "jupyterhub-singleus…"   45 seconds ago      Exited (1) 45 seconds ago                       jupyter-lidongze0629

I want to know the reason for timeout error or some method to debug why container start failed.

Thanks.

welcome commented

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.
welcome
You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

minrk commented

The docs mention that the jupyterhub in the image should match the one in your installation. 0.7.2 is very old, and unlikely what you are deploying. Make sure the pip3 install in the image is getting the version of jupyterhub you are using (likely 1.3.0).

#410 updates the docs to have more recent values if you copy/paste, but you will always need to check the version of jupyterhub inside and outside the container and make sure they match.

  • 🎉2
  • ❤️1