500 : Internal Server Error. The 'ip' trait of a Server instance must be a unicode string

Question

500 : Internal Server Error. The 'ip' trait of a Server instance must be a unicode string

KamikazeRaven opened this issue 6 years ago · 10 comments

Hello De Feo.

I followed your tutorial on https://opendreamkit.org/2018/10/17/jupyterhub-docker/
It is a very good tutorial. I think it is the only one so far that aims to help people to set up small size jupyterhub with docker container isolation.

I spend a whole day trying to make this work on the virtual machine my company assigned to me. Unfortunately, I am stuck at the very last step.

I can fire up the jupyterhub and I am able to login as an admin. However, no matter what I do, I am not able to access the single-user container that suppose to be spawn for a user.
I always get the following message after login to my hub. I'll get the same message if I try to start the single-user container at the admin page.

500 : Internal Server Error

Failed to start your server on the last attempt. Please contact admin if the issue persists.

You can try restarting your server from the home page.

here is the log from the jupyterhub:

jupyterhub_masterhub | [I 2019-04-03 21:17:22.603 JupyterHub log:158] 200 GET /hub/login (@192.168.2.27) 42.10ms
jupyterhub_masterhub | 21:21:48.369 - info: [ConfigProxy] 200 GET /api/routes
jupyterhub_masterhub | [I 2019-04-03 21:21:48.375 JupyterHub proxy:301] Checking routes
jupyterhub_masterhub | [I 2019-04-03 21:22:07.292 JupyterHub base:499] User logged in: onetech
jupyterhub_masterhub | [I 2019-04-03 21:22:07.310 JupyterHub log:158] 302 POST /hub/login?next= -> /user/onetech/ (onetech@192.168.2.27) 444.46ms
jupyterhub_masterhub | [I 2019-04-03 21:22:07.382 JupyterHub log:158] 302 GET /user/onetech/ -> /hub/user/onetech/ (@192.168.2.27) 2.47ms
jupyterhub_masterhub | [E 2019-04-03 21:22:07.460 JupyterHub base:1001] Preventing implicit spawn for onetech because last spawn failed: The 'ip' trait of a Server instance must be a unicode string, but a value of None <class 'NoneType'> was specified.
jupyterhub_masterhub | [E 2019-04-03 21:22:07.462 JupyterHub web:1670] Uncaught exception GET /hub/user/onetech/ (192.168.2.27)
jupyterhub_masterhub |     HTTPServerRequest(protocol='http', host='192.168.2.182:443', method='GET', uri='/hub/user/onetech/', version='HTTP/1.1', remote_ip='192.168.2.27')
jupyterhub_masterhub |     Traceback (most recent call last):
jupyterhub_masterhub |       File "/opt/conda/lib/python3.6/site-packages/tornado/web.py", line 1592, in _execute
jupyterhub_masterhub |         result = yield result
jupyterhub_masterhub |       File "/opt/conda/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 1003, in get
jupyterhub_masterhub |         raise copy.copy(exc).with_traceback(exc.__traceback__)
jupyterhub_masterhub |       File "/opt/conda/lib/python3.6/site-packages/tornado/web.py", line 1592, in _execute
jupyterhub_masterhub |         result = yield result
jupyterhub_masterhub |       File "/opt/conda/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 1052, in get
jupyterhub_masterhub |         await self.spawn_single_user(user)
jupyterhub_masterhub |       File "/opt/conda/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 705, in spawn_single_user
jupyterhub_masterhub |         timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future
jupyterhub_masterhub |       File "/opt/conda/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 626, in finish_user_spawn
jupyterhub_masterhub |         await spawn_future
jupyterhub_masterhub |       File "/opt/conda/lib/python3.6/site-packages/jupyterhub/user.py", line 489, in spawn
jupyterhub_masterhub |         raise e
jupyterhub_masterhub |       File "/opt/conda/lib/python3.6/site-packages/jupyterhub/user.py", line 420, in spawn
jupyterhub_masterhub |         server.ip = urlinfo.hostname
jupyterhub_masterhub |       File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 585, in __set__
jupyterhub_masterhub |         self.set(obj, value)
jupyterhub_masterhub |       File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 559, in set
jupyterhub_masterhub |         new_value = self._validate(obj, value)
jupyterhub_masterhub |       File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 591, in _validate
jupyterhub_masterhub |         value = self.validate(obj, value)
jupyterhub_masterhub |       File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 2054, in validate
jupyterhub_masterhub |         self.error(obj, value)
jupyterhub_masterhub |       File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 625, in error
jupyterhub_masterhub |         raise TraitError(e)
jupyterhub_masterhub |     traitlets.traitlets.TraitError: The 'ip' trait of a Server instance must be a unicode string, but a value of None <class 'NoneType'> was specified.
jupyterhub_masterhub |
jupyterhub_masterhub | [E 2019-04-03 21:22:07.474 JupyterHub log:150] {
jupyterhub_masterhub |       "X-Real-Ip": "192.168.2.27",
jupyterhub_masterhub |       "X-Forwarded-Server": "a58a9b87006f",
jupyterhub_masterhub |       "X-Forwarded-Proto": "http,http",
jupyterhub_masterhub |       "X-Forwarded-Port": "443,443",
jupyterhub_masterhub |       "X-Forwarded-Host": "192.168.2.182:443",
jupyterhub_masterhub |       "X-Forwarded-For": "192.168.2.27,::ffff:172.19.0.2",
jupyterhub_masterhub |       "Upgrade-Insecure-Requests": "1",
jupyterhub_masterhub |       "Referer": "http://192.168.2.182:443/hub/login",
jupyterhub_masterhub |       "Dnt": "1",
jupyterhub_masterhub |       "Cookie": "jupyterhub-hub-login=\"2|1:0|10:1554326527|20:jupyterhub-hub-login|44:YjE4OGRjOTRiMDQ1NDIxNjhjZjNlOGQxNmNiOGRmNzI=|c1946b6d8c802ea748ff93bfe30614fad7006524f4b307f577559440451b4f00\"; _xsrf=2|bb5037f2|ed9eb5d07c019fa2dd54a639bddf9d4d|1554211774; redirect=1; username-192-168-2-182-8888=\"2|1:0|10:1554304101|27:username-192-168-2-182-8888|44:ZDkxMmZhOGMyYjU1NDNmNzg3NmYxYjNjNjhhZTYyMWE=|15bac1b6b77fcbd4569c7bddc36ce08ad5f17216d47edfa83193e739e72927db\"; jupyterhub-session-id=9bd8168254ea4222bc06dd281394fe5f",
jupyterhub_masterhub |       "Accept-Language": "en-US,en;q=0.5",
jupyterhub_masterhub |       "Accept-Encoding": "gzip, deflate",
jupyterhub_masterhub |       "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
jupyterhub_masterhub |       "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0",
jupyterhub_masterhub |       "Host": "192.168.2.182:443",
jupyterhub_masterhub |       "Connection": "close"
jupyterhub_masterhub |     }

This is what my docker-compose.yml looks like. I used your basic structure, but I did not have the authentication part because my jupyterhub will be very small. I do not have an auth server. My jupyterhub will be used internally at least for now. I suspect this might be the source of the problem. I chose to use NativeAuthenticator instead. I changed the container_name to jupyterhub_masterhub because for some reason the name jupyterhub seems cause conflict. For the docker jupyer image, I chose the data science notebook, which is one of the options from this page https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html

version: '3'

services:
  jupyterhub:
    build: jupyterhub
    image: jupyterhub_img
    container_name: jupyterhub_masterhub
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - jupyterhub_data:/srv/jupyterhub
    environment:
      DOCKER_JUPYTER_IMAGE: jupyter/datascience-notebook:65761486d5d3
      DOCKER_NETWORK_NAME: ${COMPOSE_PROJECT_NAME}_default
      HUB_IP: jupyterhub_masterhub
    labels:
      - "traefik.enable=true"
      - "traefik.frontend.rule=Host:192.168.2.182"
    restart: on-failure

  jupyterlab:
    build: jupyterlab
    image: jupyter/datascience-notebook:65761486d5d3
    container_name: jupyterlab-datascience
    network_mode: none
    command: echo

  reverse-proxy:
    image: traefik
    container_name: reverse-proxy
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - ./reverse-proxy/traefik.toml:/etc/traefik/traefik.toml
      - /root/thehub/reverse-proxy/certs:/root/thehub/reverse-proxy/certs
      - /var/run/docker.sock:/var/run/docker.sock
    restart: on-failure

volumes:
  jupyterhub_data:

This is my .env file
COMPOSE_PROJECT_NAME=jupyterhub_masterhub

Here is my jupyterhub_config.py. Mine is quite different from yours because I wanted to use the simpler Native Authenticator.

import os
### Generic
c.JupyterHub.admin_access = True
c.Spawner.default_url = '/lab'

#AUTHENTICATION
#c.JupyterHub.authenticator_class = 'firstuseauthenticator.FirstUseAuthenticator'
#c.Authenticator.open_signup = True
##dummy auth testing only
#c.JupyterHub.authenticator_class = 'dummyauthenticator.DummyAuthenticator'
#c.DummyAuthenticator.password = "your strong password" #set global password

#c.JupyterHub.authenticator_class = 'jwtauthenticator.jwtauthenticator.JSONWebTokenLocalAuthenticator'


c.JupyterHub.authenticator_class = 'nativeauthenticator.NativeAuthenticator'

#c.LocalAuthenticator.create_system_users = True
c.JupyterHub.admin_access = True
c.Authenticator.admin_users = {'abe', 'root'}
c.Authenticator.whitelist = { 'abe', 'root'}




c.JupyterHub.spawner_class = 'dockerspawner.DockerSpawner'
c.DockerSpawner.image = os.environ['DOCKER_JUPYTER_IMAGE']
c.DockerSpawner.network_name = os.environ['DOCKER_NETWORK_NAME']

#c.Spawner.ip = '127.0.0.1'
#c.DockerSpawner.use_internal_ip = True
#network_name = 'bridge'
#c.DockerSpawner.network_name = network_name

# See https://github.com/jupyterhub/dockerspawner/blob/master/examples/oauth/jupyterhub_config.py
c.JupyterHub.hub_ip = os.environ['HUB_IP']

# user data persistence
# see https://github.com/jupyterhub/dockerspawner#data-persistence-and-dockerspawner
notebook_dir = os.environ.get('DOCKER_NOTEBOOK_DIR') or '/home/jovyan'
c.DockerSpawner.notebook_dir = notebook_dir
c.DockerSpawner.volumes = { 'jupyterhub-user-{username}': notebook_dir }

c.JupyterHub.services = [
    {
        'name': 'cull_idle',
        'admin': True,
        'command': 'python /srv/jupyterhub/cull_idle_servers.py --timeout=3600'.split(),
    },
]

This is my traefik.toml file under reverse-proxy folder. I changed the location of the certfile and keyfile. And I changed tls to ssl because I don't have a tls certificate. I self signed ssl.

debug = false

logLevel = "ERROR"
defaultEntryPoints = ["https","http"]

# Redirect HTTP -> HTTPS, install certificates
[entryPoints]
  [entryPoints.http]
  address = ":80"
    [entryPoints.http.redirect]
    entryPoint = "https"
  [entryPoints.https]
  address = ":443"
    [entryPoints.https.ssl]
      [[entryPoints.https.ssl.certificates]]
      certFile = "/root/thehub/reverse-proxy/certs/mycert.pem"
      keyFile = "/root/thehub/reverse-proxy/certs/mykey.key"

# Activate docker API
[docker]
domain = "docker.local"
watch = true

# Activate Traefik dashboard
[api]
  [api.statistics]
  recentErrors = 10

This is my Dockerfile in jupyterhub folder. I installed quite a few authenticators. So far only the Native Authenticator worked.

FROM jupyterhub/jupyterhub:0.9.3

COPY jupyterhub_config.py .

RUN wget https://raw.githubusercontent.com/jupyterhub/jupyterhub/0.9.3/examples/cull-idle/cull_idle_servers.py

RUN pip install \
    dockerspawner==0.10.0 \
    oauthenticator==0.8.0 \
    jupyterhub-dummyauthenticator \
    jupyterhub-firstuseauthenticator \
    jupyterhub-nativeauthenticator

This is my Dokerfile in jupyterlab folder. It's very simple. I just put this one line there so when I docker-compose build it doesn't give me error
FROM jupyter/datascience-notebook:65761486d5d3

My conda-activate.sh file is exactly same as yours. I did not change anything. Frankly, I don't know the purpose of this file. I have it here because you have it.

That is pretty much everything. I've rebuilt this docker-compose thing about 20 to 30 times. I am so close to a final working environment. Please help me solve this problem.

Thank you very much.

Answer 1 · 2019-04-04T21:41:00.000Z

Hi,

I've seen this error more than once. It is unequivocally related to a network problem starting up the single-user server, but there's multiple possible causes. See for example here: #2 (comment), where the problem was a missing .env file. I had the same error on my own instance, it was due to some arcane docker daemon failure: I had to shut down every service, then restart Docker itself, then reload the services.

In your case, these two lines looks fishy. In jupyterhub:

      DOCKER_JUPYTER_IMAGE: jupyter/datascience-notebook:65761486d5d3

and in jupyterlab:

    image: jupyter/datascience-notebook:65761486d5d3

used in combination with build: jupyterlab, the goal of image: is to set the name for the image generated by the Dockerfile in the jupyterlab folder. It is weird that you choose a name jupyter/datascience-notebook:65761486d5d3 that is already taken by a different Docker image. Try changing these two to a name of your choice, e.g., my_built_jupyterlab_image.

Answer 2 · 2019-04-04T21:42:17.000Z

P.S.: conda_activate.sh is there to fix a bug with SageMath. Since you don't use SageMath, you don't need that file.

Answer 3 · 2019-04-04T23:51:33.000Z

Hi,

I've seen this error more than once. It is unequivocally related to a network problem starting up the single-user server, but there's multiple possible causes. See for example here: #2 (comment), where the problem was a missing .env file. I had the same error on my own instance, it was due to some arcane docker daemon failure: I had to shut down every service, then restart Docker itself, then reload the services.

In your case, these two lines looks fishy. In jupyterhub:
      DOCKER_JUPYTER_IMAGE: jupyter/datascience-notebook:65761486d5d3
and in jupyterlab:
    image: jupyter/datascience-notebook:65761486d5d3
used in combination with build: jupyterlab, the goal of image: is to set the name for the image generated by the Dockerfile in the jupyterlab folder. It is weird that you choose a name jupyter/datascience-notebook:65761486d5d3 that is already taken by a different Docker image. Try changing these two to a name of your choice, e.g., my_built_jupyterlab_image.

Thank you Thank you Thank you!

I thought DOCKER_JUPYTER_IMAGE: jupyter/datascience-notebook:65761486d5d3 and image: jupyter/datascience-notebook:65761486d5d3 is where I choose which image to use. If they are only used for creating a name, then I suppose I should set the image I chose in jupyterlab's Dockerfile? Put in like FROM jupyter/datascience-notebook:65761486d5d3 ?

I'll try to fix them when I get back to work tomorrow. Now I just installed a Anaconda for root user and installed jupyterhub from there. With Naive Authenticator, I am only able to authorize system users to spawn a jupyterlab. I do not know what would happen when I use the container jupyterhub created following your tutorial.

Answer 4 · 2019-04-05T23:37:52.000Z

I thought DOCKER_JUPYTER_IMAGE: jupyter/datascience-notebook:65761486d5d3 and image: jupyter/datascience-notebook:65761486d5d3 is where I choose which image to use. If they are only used for creating a name, then I suppose I should set the image I chose in jupyterlab's Dockerfile? Put in like FROM jupyter/datascience-notebook:65761486d5d3 ?

No. This is the name the built image is going to show up with when you do docker imges. You can choose whatever name you like (typically docker compose creates one for you, but here you need to control that name in order to give it to the DOCKER_JUPYTER_IMAGE variable). Check out the docker compose docs.

Answer 5 · 2019-04-05T23:50:50.000Z

Hello

Good news, I fixed the network issue. Now the jupyterhub will try to spawn the single-user jupyterlab.

I followed your advice and changed DOCKER_JUPYTER_IMAGE: and image: (under jupyterlab) to another name in docker-compose.yml

This alone did not fix the issue. I also changed the following part in jupyterhub_config.py.

network_name = 'bridge'
c.DockerSpawner.network_name = network_name

I noticed that the spawner tried to spawn the jupyterlab at another 192.168.2.x ip. I suspected that the network issue was more or less caused by this. The jupyterhub tried to log me to this ip. Which in my case, this ip should be in the network but not within my machine.

Bad news, the single-user jupyterlab failed to spawn. It timed out. I cannot remember what did the log say. I'll update this post next week when I go back to office.

Thank you so much for your help. This is jupyterhub is the fundation of my job. I currently have two projects that will be rely on it. It is also the first thing I worked on when I was hired by my company. It means a lot to me.

Answer 6 · 2019-04-08T13:13:06.000Z

Error message:

Log

jupyterhub_masterhub | [I 2019-04-08 13:10:07.057 JupyterHub log:158] 302 GET /hub/spawn -> /user/onetech/ (onetech@192.168.2.27) 52.95ms
jupyterhub_masterhub | [I 2019-04-08 13:10:07.087 JupyterHub log:158] 302 GET /user/onetech/ -> /hub/user/onetech/ (@192.168.2.27) 2.35ms
jupyterhub_masterhub | [I 2019-04-08 13:10:07.327 JupyterHub dockerspawner:706] Found existing container jupyter-onetech (id: 55f8b5d)
jupyterhub_masterhub | [I 2019-04-08 13:10:07.328 JupyterHub dockerspawner:721] Starting container jupyter-onetech (id: 55f8b5d)
jupyterhub_masterhub | [W 2019-04-08 13:10:17.159 JupyterHub base:734] User onetech is slow to become responsive (timeout=10)
jupyterhub_masterhub | [I 2019-04-08 13:10:17.159 JupyterHub base:1056] onetech is pending spawn
jupyterhub_masterhub | [W 2019-04-08 13:10:37.490 JupyterHub user:510] onetech's server never showed up at http://127.0.0.1:32771/user/onetech/ after 30 seconds. Giving up
jupyterhub_masterhub | [I 2019-04-08 13:10:37.511 JupyterHub dockerspawner:794] Stopping container jupyter-onetech (id: 55f8b5d)
jupyterhub_masterhub | [E 2019-04-08 13:10:37.604 JupyterHub gen:974] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /opt/conda/lib/python3.6/site-packages/jupyterhub/
handlers/base.py:619> exception=TimeoutError("Server at http://127.0.0.1:32771/user/onetech/ didn't respond in 30 seconds",)> after timeout
jupyterhub_masterhub |     Traceback (most recent call last):
jupyterhub_masterhub |       File "/opt/conda/lib/python3.6/site-packages/tornado/gen.py", line 970, in error_callback
jupyterhub_masterhub |         future.result()
jupyterhub_masterhub |       File "/opt/conda/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 626, in finish_user_spawn
jupyterhub_masterhub |         await spawn_future
jupyterhub_masterhub |       File "/opt/conda/lib/python3.6/site-packages/jupyterhub/user.py", line 528, in spawn
jupyterhub_masterhub |         raise e
jupyterhub_masterhub |       File "/opt/conda/lib/python3.6/site-packages/jupyterhub/user.py", line 502, in spawn
jupyterhub_masterhub |         resp = await server.wait_up(http=True, timeout=spawner.http_timeout)
jupyterhub_masterhub |       File "/opt/conda/lib/python3.6/site-packages/jupyterhub/utils.py", line 197, in wait_for_http_server
jupyterhub_masterhub |         timeout=timeout
jupyterhub_masterhub |       File "/opt/conda/lib/python3.6/site-packages/jupyterhub/utils.py", line 155, in exponential_backoff
jupyterhub_masterhub |         raise TimeoutError(fail_message)
jupyterhub_masterhub |     TimeoutError: Server at http://127.0.0.1:32771/user/onetech/ didn't respond in 30 seconds
jupyterhub_masterhub |
jupyterhub_masterhub | [I 2019-04-08 13:10:37.610 JupyterHub log:158] 200 GET /hub/api/users/onetech/server/progress (onetech@192.168.2.27) 20306.21ms

Answer 7 · 2019-04-08T15:49:25.000Z

This log is not very useful: we only see that JupyterHub cannot reach the spawned single-user server. It might be the the single-user server is not spawning, or it might be a docker network issue. Does the single user server work if you launch it by hand with docker run?

There is no point in changing this line in jupyterhub_config.py:

c.DockerSpawner.network_name = os.environ['DOCKER_NETWORK_NAME']

You should edit the corresponding environment variable DOCKER_NETWORK_NAME in docker-compose.yml, if needed, and that variable must be exactly equal to the name of the network created by Docker compose (use docker network ls to list the docker networks).

Answer 8 · 2019-04-08T15:50:16.000Z

Also, I'm going to close this issue, because this is clearly not an issue with the code. But feel free to continue the discussion here, you can still comment after the issue is closed.

Answer 9 · 2019-04-08T17:34:12.000Z

Also, I'm going to close this issue, because this is clearly not an issue with the code. But feel free to continue the discussion here, you can still comment after the issue is closed.

Thank you for all your help. You are correct. This is not an issue with the code.

I decided to delete everything and start fresh new. This time I cloned your git and modified the lines that I think needed to be changed for my use, instead of copying your code from the tutorial. To me the code should looks the same, but miraculously this time it worked perfectly.

I don't know what part caused the problem and why it was fixed by just starting new. The only advice I can give to other people is that you should clone this git and modify it accordingly.

Answer 10 · 2019-04-08T18:07:59.000Z

Glad you managed to make it work.