jupyterhub/helm-chart

Issue with a "non-none" storage type on minikube

gvacaliuc opened this issue · 3 comments

gvacaliuc commented

Getting an issue with the Helm chart on minikube when I don't specify a singleuse storage type of none:

singleuser:
  storage:
    type: none

How to reproduce:

Launch a new minikube cluster and use a Helm + JupyterHub configuration of this repo's minikube_config.yml without lines 7-8. If you start up a singleuser server (time is a bit sensitive since kubernetes will kill the pod after a number of failed restarts) and then rush to the command line, executing the following will get some similar error messages:

$ kubectl get pods
NAME                                READY     STATUS             RESTARTS   AGE
hub-deployment-2728205822-5qrqv     1/1       Running            0          37m
jupyter-dummy                       0/1       CrashLoopBackOff   1          7s
proxy-deployment-1227971824-cdg8z   1/1       Running            0          2h

$ kubectl describe pod jupyter-dummy > describe_pod_log.txt

$ kubectl logs jupyter-dummy 
/srv/venv/lib/python3.5/site-packages/IPython/paths.py:69: UserWarning: IPython parent '/home/jovyan' is not a writable location, using a temp directory.
  " using a temp directory.".format(parent))
Traceback (most recent call last):
  File "/srv/venv/bin/jupyterhub-singleuser", line 6, in <module>
    main()
  File "/srv/venv/lib/python3.5/site-packages/jupyterhub/singleuser.py", line 322, in main
    return SingleUserNotebookApp.launch_instance(argv)
  File "/srv/venv/lib/python3.5/site-packages/jupyter_core/application.py", line 267, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/srv/venv/lib/python3.5/site-packages/traitlets/config/application.py", line 657, in launch_instance
    app.initialize(argv)
  File "<decorator-gen-7>", line 2, in initialize
  File "/srv/venv/lib/python3.5/site-packages/traitlets/config/application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "/srv/venv/lib/python3.5/site-packages/notebook/notebookapp.py", line 1290, in initialize
    super(NotebookApp, self).initialize(argv)
  File "<decorator-gen-6>", line 2, in initialize
  File "/srv/venv/lib/python3.5/site-packages/traitlets/config/application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "/srv/venv/lib/python3.5/site-packages/jupyter_core/application.py", line 243, in initialize
    self.migrate_config()
  File "/srv/venv/lib/python3.5/site-packages/jupyterhub/singleuser.py", line 239, in migrate_config
    super(SingleUserNotebookApp, self).migrate_config()
  File "/srv/venv/lib/python3.5/site-packages/jupyter_core/application.py", line 169, in migrate_config
    migrate()
  File "/srv/venv/lib/python3.5/site-packages/jupyter_core/migrate.py", line 240, in migrate
    ensure_dir_exists(env['jupyter_config'])
  File "/srv/venv/lib/python3.5/site-packages/ipython_genutils/path.py", line 167, in ensure_dir_exists
    os.makedirs(path, mode=mode)
  File "/usr/lib/python3.5/os.py", line 241, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/home/jovyan/.jupyter'

So the singleuser container is running into permissions issues writing to the /home/jovyan/ directory, which is mounted on the PersistentVolumeClaim claim-dummy, I think (see the attached file from above)? More information regarding the pvcs:

$ kubectl get pvc
NAME                      STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS   AGE
claim-dummy               Bound     pvc-087fcf72-8a86-11e7-9cf9-080027a064f0   1Gi        RWO           standard       1h
hub-db-dir                Bound     pvc-5016b6c8-8a7b-11e7-9cf9-080027a064f0   1Gi        RWO           standard       3h
jupyterhub-minikube-pvc   Bound     pvc-b5108598-8a85-11e7-9cf9-080027a064f0   1Gi        RWX           standard       1h
$ kubectl describe pvc claim-dummy 
Name:		claim-dummy
Namespace:	default
StorageClass:	standard
Status:		Bound
Volume:		pvc-087fcf72-8a86-11e7-9cf9-080027a064f0
Labels:		app=jupyterhub
		heritage=jupyterhub
		hub.jupyter.org/username=dummy
Annotations:	control-plane.alpha.kubernetes.io/leader={"holderIdentity":"c627a2c9-8a7a-11e7-9cf9-080027a064f0","leaseDurationSeconds":15,"acquireTime":"2017-08-26T17:43:13Z","renewTime":"2017-08-26T17:43:15Z","lea...
		pv.kubernetes.io/bind-completed=yes
		pv.kubernetes.io/bound-by-controller=yes
		volume.beta.kubernetes.io/storage-provisioner=k8s.io/minikube-hostpath
Capacity:	1Gi
Access Modes:	RWO
Events:		<none>
$ kubectl describe pv pvc-087fcf72-8a86-11e7-9cf9-080027a064f0 
Name:		pvc-087fcf72-8a86-11e7-9cf9-080027a064f0
Labels:		<none>
Annotations:	hostPathProvisionerIdentity=c627a242-8a7a-11e7-9cf9-080027a064f0
		pv.kubernetes.io/provisioned-by=k8s.io/minikube-hostpath
		volume.beta.kubernetes.io/storage-class=standard
StorageClass:	standard
Status:		Bound
Claim:		default/claim-dummy
Reclaim Policy:	Delete
Access Modes:	RWO
Capacity:	1Gi
Message:	
Source:
    Type:	HostPath (bare host directory volume)
    Path:	/tmp/hostpath-provisioner/pvc-087fcf72-8a86-11e7-9cf9-080027a064f0
Events:		<none>
yuvipanda commented

It looks like hostPath provisioner in minikube should respect fsGid set... I'll check upstream if there is a way to configure that.

Thank you for reporting!

minrk commented

I've worked around this by making hostpath volumes world-writable by default (wildly insecure, if it weren't in a local VM just for testing):

minikube ssh
# make existing volumes world-writable
sudo chmod -R a+rwX /tmp/hostpath-provisioner/
# make the default for new volumes world-writable as well
sudo setfacl -d -m m::rwX /tmp/hostpath-provisioner/