jupyter-fs abruptly stopped working
Cocopyth opened this issue · 10 comments
Describe the bug
I've been working with jupyter-fs since a while but it recently stopped functioning. Without any clear reason.
I now get the following error from the serverApp
wrote error: "file or directory '/a8385985:' does not exist"
followed by
404 GET /api/contents/a8385985%3A?1665515326418
File browsing doesn't work anymore and the jupyter-fs widgets are fully blank.
To Reproduce
Here is the output of conda list
# Name Version Build Channel _libgcc_mutex 0.1 main _openmp_mutex 5.1 1_gnu anyio 3.6.1 pypi_0 pypi appdirs 1.4.4 pypi_0 pypi argon2-cffi 21.3.0 pypi_0 pypi argon2-cffi-bindings 21.2.0 pypi_0 pypi attrs 22.1.0 pypi_0 pypi babel 2.10.3 pypi_0 pypi backcall 0.2.0 pypi_0 pypi beautifulsoup4 4.11.1 pypi_0 pypi bleach 5.0.1 pypi_0 pypi boto3 1.24.89 pypi_0 pypi botocore 1.27.89 pypi_0 pypi ca-certificates 2022.07.19 h06a4308_0 certifi 2022.9.24 py37h06a4308_0 cffi 1.15.1 pypi_0 pypi charset-normalizer 2.1.1 pypi_0 pypi debugpy 1.6.3 pypi_0 pypi decorator 5.1.1 pypi_0 pypi defusedxml 0.7.1 pypi_0 pypi entrypoints 0.4 pypi_0 pypi fastjsonschema 2.16.2 pypi_0 pypi fs 2.4.16 pypi_0 pypi fs-s3fs 1.1.1 pypi_0 pypi fs-smbfs 1.0.5 pypi_0 pypi idna 3.4 pypi_0 pypi importlib-metadata 5.0.0 pypi_0 pypi importlib-resources 5.10.0 pypi_0 pypi ipykernel 6.16.0 pypi_0 pypi ipython 7.34.0 pypi_0 pypi ipython-genutils 0.2.0 pypi_0 pypi jedi 0.18.1 pypi_0 pypi jinja2 3.1.2 pypi_0 pypi jmespath 1.0.1 pypi_0 pypi json5 0.9.10 pypi_0 pypi jsonschema 4.16.0 pypi_0 pypi jupyter-client 7.4.1 pypi_0 pypi jupyter-core 4.11.1 pypi_0 pypi jupyter-fs 0.3.1 pypi_0 pypi jupyter-server 1.21.0 pypi_0 pypi jupyterlab 3.4.8 pypi_0 pypi jupyterlab-pygments 0.2.2 pypi_0 pypi jupyterlab-server 2.15.2 pypi_0 pypi ld_impl_linux-64 2.38 h1181459_1 libffi 3.3 he6710b0_2 libgcc-ng 11.2.0 h1234567_1 libgomp 11.2.0 h1234567_1 libstdcxx-ng 11.2.0 h1234567_1 markupsafe 2.1.1 pypi_0 pypi matplotlib-inline 0.1.6 pypi_0 pypi mistune 2.0.4 pypi_0 pypi nbclassic 0.4.5 pypi_0 pypi nbclient 0.7.0 pypi_0 pypi nbconvert 7.2.1 pypi_0 pypi nbformat 5.7.0 pypi_0 pypi ncurses 6.3 h5eee18b_3 nest-asyncio 1.5.6 pypi_0 pypi notebook 6.4.12 pypi_0 pypi notebook-shim 0.1.0 pypi_0 pypi openssl 1.1.1q h7f8727e_0 packaging 21.3 pypi_0 pypi pandocfilters 1.5.0 pypi_0 pypi parso 0.8.3 pypi_0 pypi pexpect 4.8.0 pypi_0 pypi pickleshare 0.7.5 pypi_0 pypi pip 22.2.2 py37h06a4308_0 pkgutil-resolve-name 1.3.10 pypi_0 pypi prometheus-client 0.14.1 pypi_0 pypi prompt-toolkit 3.0.31 pypi_0 pypi psutil 5.9.2 pypi_0 pypi ptyprocess 0.7.0 pypi_0 pypi pyasn1 0.4.8 pypi_0 pypi pycparser 2.21 pypi_0 pypi pygments 2.13.0 pypi_0 pypi pyparsing 3.0.9 pypi_0 pypi pyrsistent 0.18.1 pypi_0 pypi pysmb 1.2.8 pypi_0 pypi python 3.7.13 h12debd9_0 python-dateutil 2.8.2 pypi_0 pypi pytz 2022.4 pypi_0 pypi pyzmq 24.0.1 pypi_0 pypi readline 8.1.2 h7f8727e_1 requests 2.28.1 pypi_0 pypi s3transfer 0.6.0 pypi_0 pypi send2trash 1.8.0 pypi_0 pypi setuptools 63.4.1 py37h06a4308_0 six 1.16.0 pypi_0 pypi sniffio 1.3.0 pypi_0 pypi soupsieve 2.3.2.post1 pypi_0 pypi sqlite 3.39.3 h5082296_0 terminado 0.16.0 pypi_0 pypi tinycss2 1.1.1 pypi_0 pypi tk 8.6.12 h1ccaba5_0 tomli 2.0.1 pypi_0 pypi tornado 6.2 pypi_0 pypi traitlets 5.4.0 pypi_0 pypi typing-extensions 4.4.0 pypi_0 pypi urllib3 1.26.12 pypi_0 pypi wcwidth 0.2.5 pypi_0 pypi webencodings 0.5.1 pypi_0 pypi websocket-client 1.4.1 pypi_0 pypi wheel 0.37.1 pyhd3eb1b0_0 xz 5.2.6 h5eee18b_0 zipp 3.9.0 pypi_0 pypi zlib 1.2.12 h5eee18b_3
I suspected this could have been due to incorrect jupyter config files so here is the output of jupyter --path
config: /gpfs/home2/cbisot/.jupyter /home/cbisot/.local/etc/jupyter /home/cbisot/miniconda3/envs/amftrack/etc/jupyter /usr/local/etc/jupyter /etc/jupyter data: /gpfs/home2/cbisot/.local/share/jupyter /home/cbisot/.local/share/jupyter /home/cbisot/miniconda3/envs/amftrack/share/jupyter /usr/local/share/jupyter /usr/share/jupyter runtime: /gpfs/home2/cbisot/.local/share/jupyter/runtime
Here is my jupyter_server_config.json (I've put directly the path in here thinking that it could have an impact).
{ "ServerApp": { "contents_manager_class": "jupyterfs.metamanager.MetaManager", "jpserver_extensions": { "jupyterfs.extension": true } }, "Jupyterfs": { "resources": [ { "name": "local_test", "url": "osfs:///home/cbisot" } ] } }
and jupyter_notebook_config.json
{ "NotebookApp": { "password": "******", "contents_manager_class": "jupyterfs.metamanager.MetaManager" } }
I've tried on another linux machine and still get the same issue.
Goot to know that I access the jupyterlab session remotely using a ssh tunnel.
I'm not sure this is an easily reproducible issue but after trying for quite some time now I was not able to fix it.
@Cocopyth We faced the same problem few days ago. What I figured out is we need to set ContentsManager.allow_hidden to True in jupyter_server_config.json.
Seems like exception is popping from here and there it is effectively checking if the path is behind the root directory of the notebook. For instance, if the root directory of the notebook is /home/user/jupyter/notebooks and if we set jupyterfs path as /home/user, is_hidden method of content manager will return True. So, if we do not have allow_hidden set to True, jupyterfs returns 404 response saying path not found.
Try setting ContentsManager.allow_hidden to True and see if the error still exists. Good luck!!
The above solution worked for me, pasting my full config below:
{
"ServerApp": {
"contents_manager_class": "jupyterfs.metamanager.MetaManager",
"jpserver_extensions": {
"jupyterfs.extension": true
}
},
"ContentsManager": {
"allow_hidden" : true
}
}
This fix causes the view to look weird however.
Is there a solution that doesn’t distort the view?
@Amichayg This should not affect the frontend UI. Could you paste a screenshot to better understand what is the issue?
Do note this is my first time using juptyer-fs, so i’m not familiar with the general UI.
I assumed that it utilized the same jupyter file-tree component just with a different file-system, yet this looks quite different to the general UI.
@Amichayg That is how it looks like on our deployment too. We use the main branch in our deployment. Seems like it has been changed in v0.3.0, see this comment, to this new UI.
The fix above has allowed jupyter-fs to sort of work for me, but I am unable to open folders within each resource.
@PaulKGrimes Do you still have the problem? If so, could you post JupyterLab logs and your jupyterfs config?