jupyterhub/dockerspawner

JupyterHub with SwarmSpawner SPAWNING but not connecting to Hub

ziongh opened this issue · 2 comments

ziongh commented

Bug description

JupyterHub with SwamSpawner IS spawning a docker container on the remote host, but this docker container when tries to reach to the jupyterhub master container fails.

Expected behaviour

Start Jupyter notebook on remote host and connect to jupyterhub master to proxy it.

Actual behaviour

Start Jupyter notebook on remote host but is not able to connect to jupyterhub master to proxy it.

How to reproduce

Dockerfile:

# base image: jupyterhub
# this is built by docker-compose
# from the root of this repo
FROM jupyterhub/jupyterhub
# install dockerspawner from the current repo
RUN pip install --no-cache dockerspawner
# install dummyauthenticator
RUN pip install --no-cache jupyterhub-dummyauthenticator oauthenticator
# load example configuration
ADD jupyterhub_config.py /srv/jupyterhub/jupyterhub_config.py
ADD userlist /srv/jupyterhub/userlist

docker-compose.yml

version: "3"
services:
  jupyterhub:
    env_file: .env
    build:
      context: "."
      dockerfile: "Dockerfile"
    image: swarm_jupyterhub:latest
    # This is necessary to prevent the singleton hub from using its service number as its hostname
    hostname: jupyterhub
    # Permit communication with the host's docker server
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
      - "./jupyterhub_config.py:/srv/jupyterhub/jupyterhub_config.py"
      - "./userlist:/srv/jupyterhub/userlist"
    # Ensure Hub and Notebook servers are on the same network
    networks:
      - jupyterhub_network
    environment:
      DOCKER_NETWORK_NAME: jupyterhub_network
    # Ensure that we execute on a Swarm manager
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == manager
networks:
  jupyterhub_network:
    driver: overlay
    #external: true

deploy.yml

  GNU nano 4.8                                                                                                   deployment.yml
version: "3"
services:
  hub:
    # build an image with SwarmSpawner and our jupyterhub_config.py
    env_file: .env
    image: swarm_jupyterhub:latest
    # mount the docker socket
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
      - "./jupyterhub_config.py:/srv/jupyterhub/jupyterhub_config.py"
    networks:
      - jupyterhub_network
    ports:
      - "8000:8000"
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == manager
networks:
  jupyterhub_network:
    driver: overlay
    #external: true

.ENV

GITHUB_CLIENT_ID=<GITHUB_CLIENT_ID>
GITHUB_CLIENT_SECRET=<GITHUB_CLIENT_SECRET>
OAUTH_CALLBACK_URL=https://<URL>/jupyter/hub/oauth_callback

jupyterhub_config.py

c.JupyterHub.port = 8000
c.JupyterHub.ip = '0.0.0.0'
c.JupyterHub.hub_ip = '0.0.0.0'
c.JupyterHub.bind_url = 'http://:8000/jupyter'

from dockerspawner import SwarmSpawner

class DemoFormSpawner(SwarmSpawner):
    networks = ["swarm_jupyterhub_network"]
    network_name = "swarm_jupyterhub_network"
    extra_host_config = { 'network_mode': "swarm_jupyterhub_network" }
    remove_containers = True
    default_url = '/lab'
    jupyterhub_service_name = "swarm_hub"
    debug = True
    use_internal_ip = False
    use_internal_hostname = False

    #extra_host_config.update({
    #  "extra_hosts": {
    #    hostName:hostIp
    #  }
    #})

    def _options_form_default(self):
        default_stack = "jupyter/minimal-notebook"
        return """
        <label for="stack">Select your desired stack</label>
        <select name="stack" size="1">
        <option value="jupyter/r-notebook">R: </option>
        <option value="jupyter/tensorflow-notebook">Tensorflow: </option>
        <option value="jupyterhub/singleuser">Datascience: </option>
        <option value="jupyter/all-spark-notebook">Spark: </option>
        </select>
        """.format(stack=default_stack)

    def options_from_form(self, formdata):
        options = {}
        options['stack'] = formdata['stack']
        container_image = ''.join(formdata['stack'])
        print("SPAWN: " + container_image + " IMAGE" )
        self.image = container_image
        return options

c.JupyterHub.spawner_class = DemoFormSpawner


c.JupyterHub.authenticator_class = 'oauthenticator.GitHubOAuthenticator'

c.Authenticator.whitelist = whitelist = set()
c.Authenticator.admin_users = admin = set()

import os

join = os.path.join
here = os.path.dirname(__file__)
with open(join(here, 'userlist')) as f:
    for line in f:
        if not line:
            continue
        parts = line.split()
        name = parts[0]
        whitelist.add(name)
        if len(parts) > 1 and parts[1] == 'admin':
            admin.add(name)

c.GitHubOAuthenticator.oauth_callback_url = os.environ['OAUTH_CALLBACK_URL']
  1. Build Docker Image: docker-compose build
  2. Run Swarm Stack: docker stack deploy -c deployment.yml swarm
    2.1. This will start the jupyterHub Server
  3. Login into the GitHub Auth
  4. Ask for a Kernel

After doing this the Spawner will communicate with the remote host and if you keep checking docker ps you shall see for a brief moment the notebook starting and closing.

I was able to connect to get the following log from this container between the start and End of it's life:

Executing the command: jupyterhub-singleuser --ip=0.0.0.0 --port=8888 --SingleUserNotebookApp.default_url=/lab --debug
[D 2021-02-03 17:19:49.307 SingleUserNotebookApp application:164] Searching ['/home/jovyan', '/home/jovyan/.jupyter', '/opt/conda/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2021-02-03 17:19:49.307 SingleUserNotebookApp application:730] Looking for jupyter_config in /etc/jupyter
[D 2021-02-03 17:19:49.307 SingleUserNotebookApp application:730] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2021-02-03 17:19:49.307 SingleUserNotebookApp application:730] Looking for jupyter_config in /opt/conda/etc/jupyter
[D 2021-02-03 17:19:49.307 SingleUserNotebookApp application:730] Looking for jupyter_config in /home/jovyan/.jupyter
[D 2021-02-03 17:19:49.307 SingleUserNotebookApp application:730] Looking for jupyter_config in /home/jovyan
[D 2021-02-03 17:19:49.308 SingleUserNotebookApp application:730] Looking for jupyter_notebook_config in /etc/jupyter
[D 2021-02-03 17:19:49.308 SingleUserNotebookApp application:752] Loaded config file: /etc/jupyter/jupyter_notebook_config.py
[D 2021-02-03 17:19:49.309 SingleUserNotebookApp application:730] Looking for jupyter_notebook_config in /usr/local/etc/jupyter
[D 2021-02-03 17:19:49.309 SingleUserNotebookApp application:730] Looking for jupyter_notebook_config in /opt/conda/etc/jupyter
[D 2021-02-03 17:19:49.309 SingleUserNotebookApp application:730] Looking for jupyter_notebook_config in /home/jovyan/.jupyter
[D 2021-02-03 17:19:49.309 SingleUserNotebookApp application:752] Loaded config file: /home/jovyan/.jupyter/jupyter_notebook_config.py
[D 2021-02-03 17:19:49.309 SingleUserNotebookApp application:730] Looking for jupyter_notebook_config in /home/jovyan
[W 2021-02-03 17:19:49.311 SingleUserNotebookApp configurable:190] Config option `open_browser` not recognized by `SingleUserNotebookApp`.  Did you mean `browser`?
[D 2021-02-03 17:19:49.315 SingleUserNotebookApp config_manager:96] Paths used for configuration of jupyter_notebook_config:
        /etc/jupyter/jupyter_notebook_config.json
[D 2021-02-03 17:19:49.315 SingleUserNotebookApp config_manager:96] Paths used for configuration of jupyter_notebook_config:
        /usr/local/etc/jupyter/jupyter_notebook_config.json
[D 2021-02-03 17:19:49.316 SingleUserNotebookApp config_manager:96] Paths used for configuration of jupyter_notebook_config:
        /opt/conda/etc/jupyter/jupyter_notebook_config.d/jupyterlab.json
        /opt/conda/etc/jupyter/jupyter_notebook_config.json
[D 2021-02-03 17:19:49.316 SingleUserNotebookApp config_manager:96] Paths used for configuration of jupyter_notebook_config:
        /home/jovyan/.jupyter/jupyter_notebook_config.json
[W 2021-02-03 17:19:49.984 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-02-03 17:19:49.984 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-02-03 17:19:49.984 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-02-03 17:19:49.984 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[D 2021-02-03 17:19:49.990 LabApp] Config changed: {'NotebookApp': {'open_browser': False, 'nbserver_extensions': <LazyConfigValue {'update': {'jupyterlab': True}}>}, 'ServerApp': {'ip': '0.0.0.0', 'port': 8888}, 'FileContentsManager': {'delete_to_trash': False}, 'SingleUserNotebookApp': {'ip': '0.0.0.0', 'port': 8888, 'default_url': '/lab'}, 'Application': {'log_level': 10}}
[I 2021-02-03 17:19:49.992 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.8/site-packages/jupyterlab
[I 2021-02-03 17:19:49.992 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2021-02-03 17:19:49.997 SingleUserNotebookApp mixins:557] Starting jupyterhub-singleuser server version 1.3.0
[E 2021-02-03 17:19:52.119 SingleUserNotebookApp mixins:430] Failed to connect to my Hub at http://f1cf615cf4c7:8081/jupyter/hub/api (attempt 1/5). Is it running?
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser/mixins.py", line 428, in check_hub_version
        resp = await client.fetch(self.hub_api_url)
      File "/opt/conda/lib/python3.8/site-packages/tornado/simple_httpclient.py", line 338, in run
        stream = await self.tcp_client.connect(
      File "/opt/conda/lib/python3.8/site-packages/tornado/tcpclient.py", line 265, in connect
        addrinfo = await self.resolver.resolve(host, port, af)
      File "/opt/conda/lib/python3.8/site-packages/tornado/netutil.py", line 398, in resolve
        result = await IOLoop.current().run_in_executor(
      File "/opt/conda/lib/python3.8/concurrent/futures/thread.py", line 57, in run
        result = self.fn(*self.args, **self.kwargs)
      File "/opt/conda/lib/python3.8/site-packages/tornado/netutil.py", line 382, in _resolve_addr
        addrinfo = socket.getaddrinfo(host, port, family, socket.SOCK_STREAM)
      File "/opt/conda/lib/python3.8/socket.py", line 918, in getaddrinfo
        for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
    socket.gaierror: [Errno -2] Name or service not known

Your personal set up

I'm currently running this setup in a Docker Swarm with one master in one cloud provider and on worker on another cloud provider.

Considerations

One thing that caught my eyes in the logs was this line:

Failed to connect to my Hub at http://f1cf615cf4c7:8081/jupyter/hub/api

It is considering as the hostname (f1cf615cf4c7), which is the docker Container ID of the jupyterhub master.

Probably this host is not able to resolve this hostname.

welcome commented

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.
welcome
You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

minrk commented

Failed to connect to my Hub at http://f1cf615cf4c7:8081/jupyter/hub/api

Indeed, c.JupyterHub.hub_connect_ip sets the connection host, which defaults to hostname if none is given. You should be able to use the service name:

c.JupyterHub.hub_connect_ip = 'hub'

and the single-user containers will be able to connect.

  • 🚀1