Jupyterlite demo

Question

Jupyterlite demo

psychemedia opened this issue 2 years ago · 33 comments

The simplest way to create a Jupyterlite demo requires access to a Python package on PyPi, or a prebuilt wheel that can be downloaded from the repo.

You can build a wheel by running pip3 wheel . in the root directory. (It's perhaps most convenient to create a Github Action to create the wheel on a repo push and then commit it into the repo.)

`

Answer 1 · 2022-06-16T11:32:04.000Z

Ah, just realised I can build and ship a local wheel as part of the distribution:

python -m pip wheel .
mkdir -p pypi
cp jupyter_splitview*.whl pypi/

Answer 2 · 2022-06-16T11:46:53.000Z

Okay - running into dependency issues arising from package versions installed in pyodide.

ValueError: Requested 'ipykernel>=6.13.0', but ipykernel==6.9.2 is already installed

I relaxed the pinning on ipykernel in the pyproject.toml file but the higher version may be being asserted elsewhere and I don't have the time to go chasing version conflicts right now.

For reference, this is the action I'm using:

name: Build and Deploy Jupyterlite Demo

on:
  push:
    branches:
      - main
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Setup Python
        uses: actions/setup-python@v2
        with:
          python-version: 3.9
      - name: Install the dependencies
        run: |
          python -m pip install -r .binder/requirements.txt
          python -m pip install -r .jupyterlite/requirements.txt
          python -m pip wheel .
          mkdir -p pypi
          cp jupyter_splitview*.whl pypi/
      - name: Build the JupyterLite site
        run: |
          mkdir -p content
          cp README.md content
          cp example_notebook.ipynb content
          jupyter lite build --contents ./content
      - name: Upload (dist)
        uses: actions/upload-artifact@v2
        with:
          name: jupyterlite-demo-dist-${{ github.run_number }}
          path: ./_output

  deploy:
    needs: [build]
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2.3.1
      - uses: actions/download-artifact@v2
        with:
          name: jupyterlite-demo-dist-${{ github.run_number }}
          path: ./dist
      - name: Deploy
        uses: JamesIves/github-pages-deploy-action@4.1.3
        with:
          branch: gh-pages
          folder: dist

It also pulls on a file .jupyterlite/requirements.txt:

#------ JupyerLite install -------
jupyterlite==0.1.0b8

#------ JupyerLab install -------
ipywidgets>=7.7,<8
jupyterlab~=3.3.0
jupyterlab-language-pack-fr-FR
jupyterlab-language-pack-zh-CN

#------- Demo install -------
# Reuse .binder/requirements.txt

You'd also need to activate Github Pages to build from root on gh-pages branch.

In the notebook, you'd need to:

import micropip
await micropip.install("jupyter-splitview")

Or in a guarded way for a generic notebook:

import platform as p

if p.machine()=="wasm32":
    import micropip
    await micropip.install("jupyter-splitview")

You should then be able to open into the demo notebook w/ a URL of the form:

https://kolibril13.github.io/jupyter-splitview/lab/index.html?path=example_notebook.ipynb

Answer 3 · 2022-06-16T11:58:09.000Z

Wow, this is really exciting, thank you a lot for writing down this how-to guide, I will right now try to incorporate this into the repo.

Answer 4 · 2022-06-16T13:02:23.000Z

I've now setup Jupyter lab after your draft, this would be really amazing to get working! :)
Also as expected, I get the error:
ValueError: Requested 'ipykernel>=6.13.0', but ipykernel==6.9.2 is already installed
when running

import micropip
await micropip.install("jupyter-splitview")

As an experiment, I've even fixed the kernel version here:
https://github.com/kolibril13/jupyter-splitview/blob/41948220b3768716074503b5dde46bac477d42fe/pyproject.toml#L15
but this does not change anything.
Maybe because the widget gets installed from pypi and not from the repo?
Probably something similar to this? (this does not work)

pip install git+https://github.com/kolibril13/jupyter-splitview.git@main

Answer 5 · 2022-06-16T13:52:48.000Z

I'd be tempted to remove the following from the dependencies unless you are absolutely sure they are required at a particular minimum version level.

ipython = ">=7.0.0"
ipykernel = "6.9.2"

I suspect that ipython might also set up an ipykernel dependency. (Check dependencies with eg pip show ipython, though I don't know if that shows version requirements.)

I'm not sure what the Pillow version is in pyodide but that might cause an issue too. Run (help("packages") in a code cell to list installed package, although I forget whether it includes the version number.

Answer 6 · 2022-06-16T14:03:43.000Z

Thanks for these instructions.
I'll try that out in the evening.
Is there any way that I can access the jupyter-splitview package from this repo from JupyterLite without downloading it from pip?
Making always a pip release for each debugging step seems to be a bit overkill.

Answer 7 · 2022-06-16T14:21:02.000Z

In the action, I build a wheel and put it in the pypi directory; this is copied into the gh-pages distribution and the micropip resolver picks it up from there.

Answer 8 · 2022-06-16T14:36:00.000Z

Alright, thanks for the explanation.
Later today I will try to make this work.

Answer 9 · 2022-06-16T20:05:22.000Z

After one hour of tinkering, this is my update:

The JupyterLite environment can now be accessed at https://kolibril13.github.io/jupyter-splitview
There is now a notebook called jupyterlite_debugging_notebook.ipynb that shows all installed packages with micropip.list(). The package ipykernel has here the version 6.9.2.
micropip.install("jupyter-splitview") throws an error. In both cases:
- Case 1 (this is how the dependency is also in the latest PyPi release version):
  https://github.com/kolibril13/jupyter-splitview/blob/2a6f1bed052d706794193d937d35e36a10830c0f/pyproject.toml#L15-L16
- Case 2 (here, I've set the lower bound to very low versions, so that ipykernel can be definitely be lower than 6.9.2)
  https://github.com/kolibril13/jupyter-splitview/blob/6024553ccdebbba0ba5ac9bf39016b1024aaae79/pyproject.toml#L15-L16

Thoughts:
I have no idea, why this request Requested 'ipykernel>=6.13.0' line still comes.
I am really not a version conflict expert, but maybe it's possible to solve this by not using pip, but poetry for building the package?
So here
https://github.com/kolibril13/jupyter-splitview/blob/6024553ccdebbba0ba5ac9bf39016b1024aaae79/.github/workflows/JupyterLite.yml#L23
something like this instead?
https://github.com/kolibril13/jupyter-splitview/blob/6024553ccdebbba0ba5ac9bf39016b1024aaae79/.github/workflows/pypi-publish.yml#L23-L27
Fun fact: the last few lines are from another action that generates an automatic PyPi release when I create a new tag on the main branch with the pattern 'v*..' , e.g. v.0.0.7

@psychemedia : In case that you are interested in proceeding on this (I'd be super happy if you would), I just invited you to become a collaborator in this project. For debugging pyproject.toml, the GitHub workflows and the debugging notebook, feel free to directly push on the main branch. As this project is still very small, and GitHub pages does not work nice with pull requests, developing directly on the main branch is in my opinion the fastest way to try to make this work.

Answer 10 · 2022-06-17T08:26:19.000Z

The pip install route seems happy to work with the .toml file.

I don't use poetry so have no idea how it resolves things.

If you remove the dependency on ipython and ipykernel, in my local install it builds under pip without the problematically versioned ipython package, at least according to pipdeptree.

Answer 11 · 2022-06-17T08:32:45.000Z

I also thought that the micropip install would pull from the pypi wheel in the JupyterLite distribution, but instead it seems to pull from PyPi. I don't have time to try to fix my misunderstanding of how that works atm - got a shed load of marking to do and a handover deadline to meet...!

Answer 12 · 2022-06-17T08:43:04.000Z

Okay, then I will make one test by releasing a PyPi version without the ipython and ipykernel dependency and will just see if JupyterLite and the local installed package will still work. All the best for your deadline, and thank you so much for dedicating your time to this project, even if there are so many other things around. See you around whenever you have free time again for Jupyter widget stuff :)

Answer 13 · 2022-06-17T13:20:01.000Z

Hmmm... I thought this might fix it — jupyterlite/jupyterlite#556 — but doesn't seem to? My build at https://ouseful-pr.github.io/jupyter-splitview/lab/index.html still loads the v7 wheel from pypi rather than the local v8 wheel I see in https://github.com/ouseful-PR/jupyter-splitview/tree/gh-pages/pypi ?

Answer 14 · 2022-06-17T14:58:10.000Z

Hmm, interesting.
I've now made a new pip release with the lower requirements, and now the installation does not fail anymore (yeeey 🎊! )
However, now there is a new error message when running the below cell.
@christopher-besch : Can you maybe have a quick look at this problem and see if there might be an easy fix?
To reproduce it, simply go to https://kolibril13.github.io/jupyter-splitview/lab/index.html and then start the jupyterlite_debugging_notebook.ipynb

%%splity

from skimage import data
from skimage.util import random_noise
import matplotlib.pyplot as plt

img = data.chelsea()
noisy_img = random_noise(img, var=0.02)

fig, ax1 = plt.subplots()
ax1.axis("off")
ax1.imshow(img)

fig, ax2 = plt.subplots()
ax2.axis("off")
ax2.imshow(noisy_img)

---------------------------------------------------------------------------
UndefinedError                            Traceback (most recent call last)
Input In [7], in <cell line: 1>()
----> 1 get_ipython().run_cell_magic('splity', '', '\nfrom skimage import data\nfrom skimage.util import random_noise\nimport matplotlib.pyplot as plt\n\nimg = data.chelsea()\nnoisy_img = random_noise(img, var=0.02)\n\nfig, ax1 = plt.subplots()\nax1.axis("off")\nax1.imshow(img)\n\nfig, ax2 = plt.subplots()\nax2.axis("off")\nax2.imshow(noisy_img)\n')

File /lib/python3.10/site-packages/IPython/core/interactiveshell.py:2358, in InteractiveShell.run_cell_magic(self, magic_name, line, cell)
   2356 with self.builtin_trap:
   2357     args = (magic_arg_s, cell)
-> 2358     result = fn(*args, **kwargs)
   2359 return result

File /lib/python3.10/site-packages/jupyter_splitview/sw_cellmagic.py:60, in SplitViewMagic.splity(self, line, cell)
     57 image_data_urls = [f"data:image/jpeg;base64,{base64.strip()}" for base64 in out_images_base64]
     59 # every juxtapose html node needs unique id
---> 60 inject_split(
     61     image_data_urls=image_data_urls,
     62     slider_position=slider_position,
     63     wrapper_height=int(widget_height)+4,
     64     height=int(widget_height),
     65 )

File /lib/python3.10/site-packages/jupyter_splitview/inject.py:32, in inject_split(image_data_urls, slider_position, wrapper_height, height)
     31 def inject_split(image_data_urls, slider_position, wrapper_height, height) -> None:
---> 32     html_code = compile_template(
     33         os.path.join((os.path.dirname(__file__)), "inject_split.html"),
     34         rnd_str=uuid.uuid1(),
     35         image_data_urls=image_data_urls,
     36         slider_position=slider_position,
     37         wrapper_height=wrapper_height,
     38         height=height,
     39     )
     40     display(HTML(html_code))
     41     # ensure to include the sources every time

File /lib/python3.10/site-packages/jupyter_splitview/inject.py:11, in compile_template(in_file, **variables)
      9 with open(f"{in_file}", "r", encoding="utf-8") as file:
     10     template = Template(file.read(), undefined=StrictUndefined)
---> 11 return template.render(**variables)

File /lib/python3.10/site-packages/jinja2/environment.py:1301, in Template.render(self, *args, **kwargs)
   1299     return self.environment.concat(self.root_render_func(ctx))  # type: ignore
   1300 except Exception:
-> 1301     self.environment.handle_exception()

File /lib/python3.10/site-packages/jinja2/environment.py:936, in Environment.handle_exception(self, source)
    931 """Exception handling helper.  This is used internally to either raise
    932 rewritten exceptions or return a rendered traceback for the template.
    933 """
    934 from .debug import rewrite_traceback_stack
--> 936 raise rewrite_traceback_stack(source=source)

File <template>:9, in top-level template code()

UndefinedError: list object has no element 0

Answer 15 · 2022-06-17T15:39:57.000Z

I'll take a look at it. Are you sure the html files are part of the wheel? If you didn't create it with Poetry they might not be included.

Answer 16 · 2022-06-17T16:14:32.000Z

I am really not sure how micropip works, but when I understand @psychemedia correctly, the micropip currently installs from pypi

My build at https://ouseful-pr.github.io/jupyter-splitview/lab/index.html still loads the [..] wheel from pypi

Therefore, I think they were build by poetry and should be included.

Answer 17 · 2022-06-17T16:55:35.000Z

Ok, the problem seems to be caused by image_data_urls not having two elements. This gets caused when there aren't two images shown. I've added an error message to main that makes this clearer. But I still don't know how is is being caused by JupyterLite.

Answer 18 · 2022-06-18T07:13:19.000Z

I'm having issues with setting up a dev environment for Jupyter Lite. I followed the instructions from the GitHub Action but my local changes don't get reflected when running the notebook. It seems like it always uses the version from PyPi.
I also don't understand why you aren't using poetry build to create the wheel. Wouldn't this be cleaner? And why can't poetry be used to install the dependencies as well?

Answer 19 · 2022-06-18T08:14:56.000Z

Not tried this — pyodide/pyodide#2731 (comment) — but it suggests the wheel could be among the distributed files in the JupyterLab environment and installed from there?

Answer 20 · 2022-06-18T08:25:23.000Z

I just had the same idea; I'm running into a CORS error:

ValueError: Couldn't fetch wheel from '0.0.0.0:8000/pypi/jupyter_splitview-0.0.9-py3-none-any.whl'.One common reason for this is when the server blocks Cross-Origin Resource Sharing (CORS).Check if the server is sending the correct 'Access-Control-Allow-Origin' header.

with this:

await micropip.install("/pypi/jupyter_splitview-0.0.9-py3-none-any.whl")

The weird thing is that fetch("/pypi/jupyter_splitview-0.0.9-py3-none-any.whl") runs without a problem in the browser console.
(I'm aware that that's not a very deployment-ready url.)

Answer 21 · 2022-06-18T09:23:06.000Z

I am not on the computer today, but just one quick thought: Maybe the script from #4 (comment) might help for debugging the Jupiter lite demo.

Answer 22 · 2022-06-18T12:51:42.000Z

I just ran that, this is the output:

out_images_base64 = []
type(data) = <class 'dict'>
type(list(data.values())[0]) = <class 'str'>
len(out_images_base64) = 0

The images are indeed missing.

Answer 23 · 2022-06-19T08:21:09.000Z

That means, the images are not displayed. That was easy to fix!
I've noted that in local notebooks, creating a created figure will automatically show, in JupyterLite, a figure has to be explicitly called by plt.show().

%%splity
import matplotlib.pyplot as plt
import numpy as np

array1 = np.full((15, 30), 10)
array2 = np.random.randint(0, 10, size=(15, 30))
fig, ax1 = plt.subplots()
ax1.imshow(array1)
plt.show()
fig, ax2 = plt.subplots()
ax2.imshow(array2)
plt.show()

So this works like charm in JupyterLite.
Thanks to the two of you, for all your input and debugging efforts.
I'll update the example notebook right now.

Answer 24 · 2022-06-19T09:16:15.000Z

Here is now the final JupyterLite implementation:
https://kolibril13.github.io/jupyter-splitview/

One thing I'd like to simplify:
The first cell

import micropip  #  only for JupyterLite
await micropip.install("jupyter-splitview")

could be relocated somewhere to the build process.
I've already added the dependency jupyter-splitview here:

https://github.com/kolibril13/jupyter-splitview/blob/2cc206ae933c53d8bbee4411e464014664ef7f7b/.jupyterlite/requirements.txt#L1-L2
But this did not work.
@psychemedia : Any ideas how to set up this dependency for JupyterLite? I think it's fine when the dependency is coming from PyPi, and is not build by GitHub actions from the latest main commit.

Answer 25 · 2022-06-19T10:36:22.000Z

I think auto/pre-running code is still an open issue; eg jupyterlite/jupyterlite#508

Answer 26 · 2022-06-19T11:08:53.000Z

Thanks for the reference. But pre-installing packages should be already possible, as also the binder requirements (NumPy, Matplotlib, etc.) are installed beforehand:

https://github.com/kolibril13/jupyter-splitview/blob/32acb75ce09d7b3af1c559f16b919fa0d25124ae/.github/workflows/JupyterLite.yml#L21

So why would that not be possible with the jupyter-splitview package?

UPDATE:
I've now experimentally added the package here, but it still does not work:

https://github.com/kolibril13/jupyter-splitview/blob/bf6e6460eef0dc7fdb42444ef4f6ee779914c2c6/.binder/requirements.txt#L4-L6

Answer 27 · 2022-06-20T16:55:25.000Z

The matplotlib package is already part of the base pyodide environment and JupyterLab extensions that are installed in the host environment are also migrated into the JupyterLite distribution.

But AFAIK, arbitrary packages that are typically end user installed must still be handled manually. @jtpio could perhaps clarify the roadmap and current best practice on this.

Answer 28 · 2022-06-20T17:25:54.000Z

Thanks for this clarification!
Maybe it could an idea to mention the packages from base pyodide environment at https://jupyterlite.readthedocs.io/en/latest/howto/python/packages.html

Regarding this project:
I think it's then better to remove this line:
https://github.com/kolibril13/jupyter-splitview/blob/bf6e6460eef0dc7fdb42444ef4f6ee779914c2c6/.github/workflows/JupyterLite.yml#L21
Furthermore this file can be re-arranged:
https://github.com/kolibril13/jupyter-splitview/blob/bf6e6460eef0dc7fdb42444ef4f6ee779914c2c6/.binder/requirements.txt#L1-L6
For binder, I remove the last line of binder/requirements.txt.
For JupyterLite, I remove the last and the first line binder/requirements.txt and insert it at the top of
https://github.com/kolibril13/jupyter-splitview/blob/bf6e6460eef0dc7fdb42444ef4f6ee779914c2c6/.jupyterlite/requirements.txt#L1-L2

Answer 29 · 2022-06-21T10:00:22.000Z

One reason for installing via . in binder requirements is that it lets you try the latest build as per the repo, rather than the latest build as per whatever package is on PyPi. If Binder builds from PyPi, you can always get access to a runnable version of the latest dev version in.

Answer 30 · 2022-06-21T10:34:48.000Z

Thanks!
It would be amazing if we could build the split-view package with GitHub Actions for JupyterLite from the repo as well!
As GithHub pages can be build from pull requests, each pull request could have their own JupyterLite environment with the corresponding package version from the pull request.
That would mean one can simultaneously compare different pull requests by opening multiple tabs in the browser and see how functionality is changing.
No more downloading the branch, opening the local IDE, starting a local JupyterServer and then start testing.
Only pressing one button and then start testing in the browser!
The GitHub action bot can even send a message into a pull request to link to the right page.
Here is a demo
Pull-request:
https://github.com/kolibril13/okapi2/pull/7
Main page:
https://kolibril13.github.io/okapi2/
Pull request demo page:
https://kolibril13.github.io/okapi2/pr-preview/pr-7/

Answer 31 · 2022-06-21T13:35:27.000Z

I have no time to chase this right now, but may have a pointer on how to install packages.. https://twitter.com/martinRenou/status/1539236698068074497

Answer 32 · 2022-06-23T09:23:04.000Z

The jupyterlite-xeus-python kernel allows the bundling of python packages as part of the distribution [docs].

Example automation script for producing JupyterLite distribution, including UI test regime: https://github.com/martinRenou/ipycanvas/blob/master/.github/workflows/main.yml

Answer 33 · 2022-06-27T14:54:14.000Z

Thanks for the link!
With the current status of the project I am already really happy, I might come back to this in a few months when I have more time for tweaking this workflow. Maybe there's even a nice poetry workflow available by then.