fonsp/PlutoUtils.jl

JupyterLab integration

fonsp opened this issue · 28 comments

fonsp commented

Help welcome!

I managed to get this working for one setup, but I am not sure how to make it more general so that other people can install it.

The repository is here:

https://github.com/fonsp/pluto-on-binder

with a demo on binder:

https://mybinder.org/v2/gh/fonsp/pluto-on-binder/master
(it's one of the options when launching a new kernel/notebook)
or you can skip the Jupyter UI and go straight to Pluto:
https://mybinder.org/v2/gh/fonsp/pluto-on-binder/master?urlpath=pluto

I also got it running as jupyterlab extension with a nice button and logo

It sets up jupyter for binder with the http server proxy extension, and installs a mini python package that is used by Jupter as an extension, available through ?urlpath=pluto.

Goals

  1. Instructions for other people who have JupyterHub/Lab (not sure which is which) running on how to add Pluto. Should plutoproxy be a registered Python package?
  2. (down the road) Integrate JupyterLab-specific UI into Pluto, like the RAM usage, a button to go back to the Lab Hub menu thingy, Pluto notebooks as a tab within Lab, instead of a full-screen window, and what else?
shashankp commented

lungben#3 is to make Pluto work with Binder, which is essentially Jupyter.
Should work in local/binder/any other env

This works right now

docker run -it -p 8080:8080 plutojl/pluto-binder:latest jupyter notebook --ip=0.0.0.0 --port=8080
fonsp commented

We can run Pluto notebooks as Lab tabs:
https://jupyter-server-proxy.readthedocs.io/en/latest/server-process.html

Also set a very high HTTP timeout (60 works)

koehlerson commented

let me know if you need the post without the teaching stuff overhead somewhere

fonsp commented

Could someone take a look at: https://github.com/illumidesk/jupyter-pluto-proxy Which Pluto version does it use? Will it be stuck on an outdated version? @jgwerner

It's listed in the official jupyter server proxy docs: https://jupyter-server-proxy.readthedocs.io/en/latest/examples.html

koehlerson commented

The repo itself or the binder? Binder used 12.4. when I tried. I think it will work ootb as long as these lines don't change for starting a Pluto.jl Server https://github.com/IllumiDesk/jupyter-pluto-proxy/blob/main/jupyter_pluto_proxy/__init__.py#L23

jgwerner commented

HI @fonsp ! First of all kudos for developing Pluto, 💯 We can update Pluto in this example repo no problem, should we just update Pluto to the latest stable release? We could also add a bot to automatically create a PR to update the version when you cut a new release. In our case, the use-case was to set up a POC since our tool uses the JupyterHub and some folks are more on the Julia side of things than Python.

@koehlerson regarding the default command, we can update that too.

As far as the perceived official status goes, we sometimes send a PR to the jupyter-server-proxy repo and/or add jupyter-server-proxy as a repo topic. Both of these were recommendations from @betatim from this thread. To avoid duplicating "official" repos, we would gladly transfer what we did to your GH account/org and help contribute to the repo in your account.

fonsp commented

@koehlerson @barche @jgwerner @GiggleLiu You have all managed to run Pluto on JupyterLab, that's great! @koehlerson even wrote a detailed guide: https://www.maximiliankoehler.de/posts/pluto-server/

Let's work together to create a simple and clear way for people to install Pluto as an extension.

I think @koehlerson has done most of the work already, but still left is:

  • It should not use a pluto-on-binder repository -- this is confusing for me (what am I maintaining?) and for others. It looks like some people have installed fonsp/pluto-on-binder as a lab extension instead of barche/pluto-on-binder. I have made a new repository https://github.com/fonsp/pluto-on-jupyterlab where you all have write access, let's make this the place to house:
    • a README (redirecting to @koehlerson 's blog) and
    • all files needed for the extension
  • Make sure that it works with the most recent version of Pluto
  • How will people update Pluto after they have installed the extension? (pip update?) This needs to be documented clearly
  • Add link to instructions to fonsp/Pluto.jl/wiki, add redirect and warning to fonsp/pluto-on-binder/README.md

And then we can close this issue!

fonsp commented

FYI, the differences between pluto-on-binder and pluto-on-jupyterlab are:

  • pluto-on-binder has all security disabled, and will have a modified HTTP proxy that is even less secure to allow unauthenticated CORS
  • pluto-on-binder does not have a logo
  • At some point, pluto-on-binder will just be a single Dockerfile to optimize the launch times even further
  • At some point, pluto-on-jupyterlab might have UI specific for jupyterlab, like a button to go back to the lab menu (feel free to make more suggestions!)
koehlerson commented

Hey Fons,
this sounds good. I pushed the setup I use on my server, which should work for the current version of Pluto.jl as well. However, I cannot confirm and I have a lot of deadlines this week. Maybe @GiggleLiu can try it? Sorry again @GiggleLiu for the wrong pip plugin link in the tutorial. I'll fix all links and commands as soon as the new repo is good to go for reproduction.

Feel free to remove the .toml and .yml files/file. I wasn't sure if they are leftovers from the fork or if they are actually needed. IMHO they can be removed, but I'm not confident enough to guarantee it.

If you are interested in the interaction of WGLMakie and such a Pluto.jl server, I can also describe in a short tutorial how admins can set this up.

  • ❤️1
barche commented

This is a very nice initiative, I'm certainly willing to describe more in detail what our setup is and integrate pluto-on-jupyterlab into our workflow. I had a quick look at @koehlerson 's nice guide, but most of it is actually about setting up jupyterhub (i.e. the multi-user server) itself, and there are many different approaches for that. We use the docker spawner, where each logged in user starts up jupyterlab in his own docker container and can also launch Pluto from there by clicking a button.

The actions needed to add Pluto are actually quite simple and just a short part, so we should maybe delimit what part exactly we want to describe as part of the Pluto documentation.

  • ❤️2
GiggleLiu commented

Hi, @fonsp and @koehlerson

I just tried this exciting new feature following @koehlerson 's nice blog. From a user perspective, I feel the blog is clearly written. So far, except the user permission issue, it works perfectly. Hope the following feedbacks can be helpful.

about the blog writting

  1. For AWS users, is it nessesary to setup static IP? I just ignored the first section, hope doing this is safe. (aws setup reference: https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html)

  2. add mkdir /usr/share/juliapackages before the chown command.

  3. The nodejs of ubuntu source is too old. This guide is helpful to me: https://tecadmin.net/install-latest-nodejs-npm-on-ubuntu/

  4. jupyterlab 2.2.8 works, however, the latest (default) 3.0.1 does not.

  5. When I add a new user without the -m option, the server errors on home folder now exist.
    sudo useradd -aG juliausers studentXX -> sudo useradd -amG juliausers studentXX. This should be double checked. The error I see is

[I 2021-01-06 18:24:05.452 JupyterHub spawner:1451] Spawning /opt/jupyterhub/bin/jupyterhub-singleuser --port=34457 --SingleUserNotebookApp.default_url=/lab 

Couldn't set CWD to /home/grubby ([Errno 2] No such file or directory: '/home/grubby') 

Traceback (most recent call last): 

  File "/opt/jupyterhub/lib/python3.6/site-packages/traitlets/traitlets.py", line 528, in get 

    value = obj._trait_values[self.name] 

KeyError: 'runtime_dir'
  1. It is helpful to tell users how to read the log: sudo journalctl -u jupyterhub

about pluto-jupyter

  1. Is there a better way to add a package. Like colab, one can freely add packages and remove it after the session dies.
  2. Is it possible to specify the sample folder so that users can see the prepared samples.
  3. I have to run chown and chmod 777 on /usr/share/juliapackages repeatedly to make sure the permissions are correct. This is the issue that makes me headache.
  • ❤️3
koehlerson commented

thanks for the input @GiggleLiu !

I think we clearly need to differ use cases. For me I want that no one adds packages except sudo users. Outside the realm of teaching it makes sense to let users add packages on their own. But, if there are multiple ones and you share a common depot path, then you need to be careful.
With the default settings, i.e. not setting the common julia depot path (delete every JULIA_DEPOT_PATH variable you've set during the tutorial on my website), packages will be installed under /home/studenXX/.julia/. So, each user has its own package directory and can do with it whatever she/he wants to do.
I do not know enough about AWS, but in general the computer where I installed all of this should have the same ip whenever I want to access it. I guess you have some sort of ip or url from aws which stays the same? Or at least the aws system redirects you always to your machine/account allocated ressources

@barche can you access with this approach the proxy pluto server? So, something like this:

user = ENV["USER"]
external_ip = "134.147.244.113/user/$user/proxy/"
port = 3389

Thanks again to everyone contributing to this!

GiggleLiu commented

Thanks for the reply.

With the default settings, i.e. not setting the common julia depot path (delete every JULIA_DEPOT_PATH variable you've set during the tutorial on my website), packages will be installed under /home/studenXX/.julia/. So, each user has its own package directory and can do with it whatever she/he wants to do.

nice!

I do not know enough about AWS, but in general the computer where I installed all of this should have the same ip whenever I want to access it. I guess you have some sort of ip or url from aws which stays the same? Or at least the aws system redirects you always to your machine/account allocated ressources.

Yes, an AWS host has a static IP and also a url. I think this is worth mention in the blog so that people can skip this sector with confidence.

jgwerner commented

@fonsp Let me know if you would like us to hand over the repo to you to ensure proper upgrades. We would also be glad to help you out here and perhaps attempt to automate creating PRs when you release a new version.

My 2 cents is that JupyterLab version 3.0 was just released as far as I know those extensions are not compatible with 1.x/2.x extensions.

fonsp commented

Whatever you think is best! You all understand this much better than I do

jgwerner commented

Whatever you think is best! You all understand this much better than I do

@fonsp That I doubt very much 😄 - I vote for passing this on to your account (or an org that you designate) and then we would be glad to contribute any/all code we feel necessary to the code in that location. What do you think?

jgwerner commented

@fonsp could you create a repo for your account for jupyter-pluto-proxy (or any other name that you would like)? With that, I can send you a PR with the legacy code-base and update the license from BSD-3 Clause (we left it with BSD for the sake of consistency with the Jupyter project) to Unlicensed so that you may add the license of your choosing.

I'll do my best to update deps/docs for the first PR to your account and then open issues in your repo for items that we have taken note of. As a side note, we can provide the step to add the jupyter-pluto-proxy to the Jupyter Server Proxy "official" examples and remove the reference to ours to avoid confusion. We're excited to help you get this started with you!

fonsp commented

Okay! But don't we already have this here? https://github.com/fonsp/pluto-on-jupyterlab/tree/master/plutoserver

It has to be clear for people what they need to install (and how they update Pluto after it is installed), so having a single repository would be helpful.

pankgeorg commented

@GiggleLiu the compatibility with 3.0.1 seems to be a problem with jupyter -- see this comment (They're waiting for extension compatibility before updating Docker stacks to 3.0.1)

  • ❤️1
jgwerner commented

Okay! But don't we already have this here? https://github.com/fonsp/pluto-on-jupyterlab/tree/master/plutoserver

It has to be clear for people what they need to install (and how they update Pluto after it is installed), so having a single repository would be helpful.

@fonsp Ah cool! I'll review that repo then and send a PR in case we see something that we have on our end that is missing from yours. Then we will archive ours and add yours as a reference in the Readme.

fonsp commented

@GiggleLiu @jgwerner @koehlerson Can we try to create a single guide that explains how to install pluto as a jupyterlab extension? I still don't know what to say when people ask me :(

barche commented

I'm currently using the unmerged PR from @pankgeorg, in the way described here: https://github.com/barche/plutohub-juliacon2021

jgwerner commented

@fonsp we are just using the jupyter-server-proxy extension and then configure it to run Pluto: https://github.com/IllumiDesk/jupyter-pluto-proxy.

johannes-fischer commented

I have been looking around for a way to share Pluto notebooks with students in a way that they can run them on binder. I have found various github repos and issues in that direction, but I couldn't find something working. Is this still work in progress, or is there a working solution for this somewhere around?

fonsp commented

@johannes-fischer You should try https://github.com/JuliaPluto/static-export-template/ ! Open an issue there if something is not working

johannes-fischer commented

@fonsp thanks, will have a look there!