Repackage Sage's cropped threejs as a pip-installable package (nbextension) jupyter-threejs-sage
mkoeppe opened this issue · 72 comments
As a follow-up to #30972, we provide a pip-installable package (nbextension) that makes (all supported versions of) the threejs-sage offline script available to a Jupyter notebook:
This allows users to set up a system Jupyter notebook in which Sage is fully functional (including offline 3d graphics), without having to manually create symlinks from a Sage installation to the system's nbextensions directory.
see also:
- https://groups.google.com/d/msg/sage-devel/mWTD0_iBwKc/wvhk-VtKBgAJ
- https://groups.google.com/d/msg/sage-support/Jaiqx-KTwM0/BVxEp1gxAwAJ
- #30296 System package information and
spkg-configureformathjax
User reports:
Depends on #30972
CC: @dimpase @kliem @paulmasson @jcamp0x2a @nbruin @slel @mjungmath @orlitzky @kiwifb @antonio-rojas @mwageringel
Component: user interface
Keywords: sd111
Author: Matthias Koeppe
Branch/Commit: u/mkoeppe/repackage_sage_s_cropped_threejs_as_a_pip_installable_package__nbextension__jupyter_threejs_sage @ 74ade58
Issue created by migration from https://trac.sagemath.org/ticket/30123
My guess is that the new install script build/pkgs/sagelib/spkg-install poisons environment variables too aggressively.
The code that installs the Jupyter kernel has moved recently to src/sage_setup/command/sage_install.py.
This should be investigated by someone who knows about the notebook.
According to Dima Pasechnik, this can be fixed by sage -b
Indeed...
That is neverthelass a bug that should be fixed.
Note that sage -b doesn't work any more: it tries to do cd "$SAGE_SRC" && $MAKE, but SAGE_SRC/Makefile.in was removed in #29411. (If you have done an incremental build, the Makefile is still there.)
Replying to @jhpalmieri:
Note that
sage -bdoesn't work any more: it tries to docd "$SAGE_SRC" && $MAKE, butSAGE_SRC/Makefile.inwas removed in #29411. (If you have done an incremental build, the Makefile is still there.)
Let's fix that in #30153.
Replying to @mkoeppe:
Replying to @jhpalmieri:
Note that
sage -bdoesn't work any more: it tries to docd "$SAGE_SRC" && $MAKE, butSAGE_SRC/Makefile.inwas removed in #29411. (If you have done an incremental build, the Makefile is still there.)Let's fix that in #30153.
As of Sage 9.2.beta7 (which includes #30153), sage -b does no longer work to fix the Jupyter issue.
FWIW, here is the content of SAGE_ROOT/local/share/jupyter/nbextensions with Sage 9.2.beta7:
lrwxrwxrwx 1 eric eric 19 août 3 15:12 jsmol -> /doesnotexist/jsmol
drwxr-xr-x 2 eric eric 4096 mai 30 17:52 jupyter-js-widgets
lrwxrwxrwx 1 eric eric 21 août 3 15:12 mathjax -> /doesnotexist/mathjax
lrwxrwxrwx 1 eric eric 21 août 3 15:12 threejs -> /doesnotexist/threejs
The doesnotexist directory does not look good...
Changing manually the symbolic links to
lrwxrwxrwx 1 eric eric 46 août 3 17:40 jsmol -> /home/eric/sage/9.2.develop/local/share/jsmol/
drwxr-xr-x 2 eric eric 4096 mai 30 17:52 jupyter-js-widgets
lrwxrwxrwx 1 eric eric 48 août 3 17:40 mathjax -> /home/eric/sage/9.2.develop/local/share/mathjax/
lrwxrwxrwx 1 eric eric 48 août 3 17:40 threejs -> /home/eric/sage/9.2.develop/local/share/threejs/
fixes the Jupyter notebook issue reported in this ticket.
Thanks for investigating this.
This is, as I thought, due to the poisoning of environment variables in the script build/pkgs/sagelib/spkg-install.
The links that created here seem to assume that jsmol, mathjax, and threejs are installed in SAGE_LOCAL.
The correct fix is to create spkg-configure.m4 scripts for these packages so that equivalent system packages can be used -- and to use specific directory variables for determining the targets for these symlinks.
In fact, it seems strange that the sagelib installation script would even install these links. Shouldn't the install script of mathjax install its symlink?
Description changed:
---
+++
@@ -1,5 +1,5 @@
https://groups.google.com/d/msg/sage-devel/mWTD0_iBwKc/wvhk-VtKBgAJ
-reported to have been broken by #29411
+broken by #29411
Hoping that someone who knows about these javascript things will work on this ticket.
for some reason ./sage -i mathjax rebuilds sagelib.
I think this ticket is moving in the wrong direction. The issue is not with the installation of JavaScript packages, it is that they all expect a link to nbextensions to exist. There is an entire install file for Jupyter that sets up the environment for JavaScript packages by a call to its update method. Is this method no longer being called after installing the notebook?
As you can see in comment 8, it is called but the paths are configured wrong.
So the notebook doesn't work because of an explicit mangling of environment variables. Why can't you just unmangle the appropriate directory so that notebook setup works? Oddly enough all of the tests in src/sage/repl/ipython_kernel/install.py pass despite the mangling.
As for using a system Three.js I would definitely advise against that. Mr.doob very often makes changes that are NOT backward compatible. The template that renders Three.js is version dependent: you can't just switch to a newer version and expect it to work. That's why I continue to extract the core of the library and package it here separately from the upstream repository.
Replying to @paulmasson:
Oddly enough all of the tests in
src/sage/repl/ipython_kernel/install.pypass despite the mangling.
The mangling happens during install, not during testing.
As for using a system Three.js I would definitely advise against that. Mr.doob very often makes changes that are NOT backward compatible. The template that renders Three.js is version dependent: you can't just switch to a newer version and expect it to work.
That's an important bit of information.
That's why I continue to extract the core of the library and package it here separately from the upstream repository.
You package it where?
Replying to @mkoeppe:
Replying to @paulmasson:
That's why I continue to extract the core of the library and package it here separately from the upstream repository.
You package it where?
If you look at the upgrade tickets, like #29809, I always attach a zip file containing a small portion of the full library. I've been doing that since I started the viewer, at first just to avoid downloading the full library. There is however a change coming in the new year, in which one of the two critical files we use will officially be removed from the library. That will necessitate either continuing the current process or modifying a local copy of the full library, a process over which we would not be guaranteed control.
Volker has been moving my zip files to the mirrors manually. Do you know if that is also the process for libraries with upstream links?
Replying to @paulmasson:
If you look at the upgrade tickets, like #29809, I always attach a zip file containing a small portion of the full library. [...]
Volker has been moving my zip files to the mirrors manually. Do you know if that is also the process for libraries with upstream links?
Thanks. I am looking at build/pkgs/threejs. Is the spkg-src script current? Is this what you actually use to generate archives?
Since the 9.1 cycle we have a better mechanism for tickets that upgrade spkgs. See https://wiki.sagemath.org/ReleaseTours/sage-9.1#For_developers-1
Replying to @mkoeppe:
Replying to @paulmasson:
As for using a system Three.js I would definitely advise against that. Mr.doob very often makes changes that are NOT backward compatible. The template that renders Three.js is version dependent: you can't just switch to a newer version and expect it to work.
OK, but is it possible to reliably test for presence of a specific version in the system?
Do you know how distribution packagers handle this? (Debian, Gentoo, ...)
Replying to @mkoeppe:
Replying to @paulmasson:
If you look at the upgrade tickets, like #29809, I always attach a zip file containing a small portion of the full library. [...]
Volker has been moving my zip files to the mirrors manually. Do you know if that is also the process for libraries with upstream links?
Thanks. I am looking at
build/pkgs/threejs. Is thespkg-srcscript current? Is this what you actually use to generate archives?
That's precisely what I used for the last update.
Replying to @mkoeppe:
Replying to @mkoeppe:
Replying to @paulmasson:
As for using a system Three.js I would definitely advise against that. Mr.doob very often makes changes that are NOT backward compatible. The template that renders Three.js is version dependent: you can't just switch to a newer version and expect it to work.
OK, but is it possible to reliably test for presence of a specific version in the system?
Do you know how distribution packagers handle this? (Debian, Gentoo, ...)
You can search for a REVISION string as is done here. This code was written to ditch installed_packages to help package managers.
On a more general note, what if a user installs a newer package of any sort that creates conflicts with Sage? Do you then force installation of the version needed by Sage?
Replying to @paulmasson:
You can search for a REVISION string as is done here. This code was written to ditch
installed_packagesto help package managers.
Thanks. So this looks at a local installation (located via the THREEJS_DIR variable)
and then picks specific versions of remote URLs.
When you make updates, do you make simultaneous changes to both sagelib and to the threejs spkg?
On a more general note, what if a user installs a newer package of any sort that creates conflicts with Sage? Do you then force installation of the version needed by Sage?
The spkg-configure mechanism of the Sage distribution inspects installed system packages. We either test for particular features of a library/program, or for a range of versions. For example, currently we accept python3 version 3.6.x and 3.7.x but nothing older nor newer. Whenever a package is found to be installed but not suitable, the Sage distribution installs its own copy.
The standard way to install notebook extensions seems to be using jupyter nbextension. I think we may want to use this (instead of the custom code in sage.repl.ipython_kernel.install).
Replying to @mkoeppe:
The standard way to install notebook extensions seems to be using
jupyter nbextension. I think we may want to use this (instead of the custom code insage.repl.ipython_kernel.install).
In many ways jupyter nbextension is for personal install - which I guess is OK for vanilla sage. But I don't really know if it is designed well enough to be handled properly by a bona fide package management system.
The only real example I have on top of my head is widgetsnbextension and it uses regular distutils to install the extension.
Replying to @mkoeppe:
Replying to @paulmasson:
You can search for a REVISION string as is done here. This code was written to ditch
installed_packagesto help package managers.Thanks. So this looks at a local installation (located via the
THREEJS_DIRvariable)
and then picks specific versions of remote URLs.When you make updates, do you make simultaneous changes to both sagelib and to the threejs spkg?
I only change threejs.
Replying to @paulmasson:
Replying to @mkoeppe:
Replying to @paulmasson:
You can search for a REVISION string as is done here. This code was written to ditch
installed_packagesto help package managers.Thanks. So this looks at a local installation (located via the
THREEJS_DIRvariable)
and then picks specific versions of remote URLs.When you make updates, do you make simultaneous changes to both sagelib and to the threejs spkg?
I only change threejs.
OK. To summarize, THREEJS_DIR is used at least twice:
(1) during installation of sagelib, to determine symlinks that will be placed in the nbextensions folder.
(2) during runtime of sagelib, to determine the threejs version to get certain remote URLs right.
Do both of them have to point to the same installation (same version) for things to work, or are they two separate, independent uses?
Also, suppose we develop a way so that users can install sagelib using pip, instead of going through the sage distribution. Would it make sense to provide a copy of whatever parts of threejs that sagelib needs as a pip-installable Python package?
Is https://github.com/jupyter-widgets/pythreejs of any relevance in this direction?
Replying to @kiwifb:
Replying to @mkoeppe:
The standard way to install notebook extensions seems to be using
jupyter nbextension. I think we may want to use this (instead of the custom code insage.repl.ipython_kernel.install).[...] I don't really know if it is designed well enough to be handled properly by a bona fide package management system.
In gentoo, are you able to use the sage.repl.ipython_kernel.install during packaging, or are you using a custom script for this?
Replying to @mkoeppe:
Replying to @paulmasson:
Replying to @mkoeppe:
Replying to @paulmasson:
You can search for a REVISION string as is done here. This code was written to ditch
installed_packagesto help package managers.Thanks. So this looks at a local installation (located via the
THREEJS_DIRvariable)
and then picks specific versions of remote URLs.When you make updates, do you make simultaneous changes to both sagelib and to the threejs spkg?
I only change threejs.
OK. To summarize,
THREEJS_DIRis used at least twice:
(1) during installation of sagelib, to determine symlinks that will be placed in thenbextensionsfolder.
(2) during runtime of sagelib, to determine the threejs version to get certain remote URLs right.
Do both of them have to point to the same installation (same version) for things to work, or are they two separate, independent uses?
Since there is only one Three.js template, they must both point to the same version.
Also, suppose we develop a way so that users can install sagelib using
pip, instead of going through the sage distribution. Would it make sense to provide a copy of whatever parts of threejs that sagelib needs as a pip-installable Python package?
Is https://github.com/jupyter-widgets/pythreejs of any relevance in this direction?
pythreejs is just a port of Three.js to python, so not relevant: one needs to get the actual JavaScript files. I don't know much about pip, but if you just mean taking my small portion of the Three.js library and delivering it some way other than through Sage, then that is certainly possible.
In Gentoo I disable instance._symlink_resources() in the update method (this is all in sage/repl/ipython_kernel/install.py so it is not run. I do these manually from the ebuild.
This bit only creates links to three things, two icons and the documentation. I mainly do them manually because I want relative links instead of absolute links and it is quite possible that at the time it is normally executed things would end up pointing to the wrong places - which would be the real killer out of the two.
It is quite an achievement that we have come down to these three little things, it used to be much more complex.
And I should add those three links are going in the sage kernel location not in nbextensions. The nbextensions are put in the exact right place (absolute links but we can't always have everything).
Replying to @paulmasson:
I don't know much about pip, but if you just mean taking my small portion of the Three.js library and delivering it some way other than through Sage, then that is certainly possible.
Yes, that's what I mean. Basically making it available on https://pypi.org/ so that users can install it with pip. (The Sage distribution would still ship a copy of it.)
I'll be happy to help with the packaging.
This would be a good step in the direction of plugging Sage (which is installed into its own venv) into a system Jupyter installation.
The package could be called jupyter-threejs-minimal or something like this. See https://pypi.org/search/?q=jupyter
Description changed:
---
+++
@@ -2,4 +2,4 @@
broken by #29411
-
+see also https://groups.google.com/d/msg/sage-support/Jaiqx-KTwM0/BVxEp1gxAwAJWhat's the situation with jsmol?
Description changed:
---
+++
@@ -2,4 +2,7 @@
broken by #29411
-see also https://groups.google.com/d/msg/sage-support/Jaiqx-KTwM0/BVxEp1gxAwAJ
+see also:
+
+- https://groups.google.com/d/msg/sage-support/Jaiqx-KTwM0/BVxEp1gxAwAJ
+- #30296 System package information and `spkg-configure` for `mathjax`Since the repackage still needs to ship with Sage to make the viewer function, this isn’t going to solve the notebook problem.
By itself, no, but it's part of the solution. I cc'd you on #30124, which is another part of the puzzle.
I'll fix the notebook in #30298, but this ticket here remains relevant.
I don't really understand the discussion here. But I remember the comment I made before in a sage-devel thread: https://groups.google.com/g/sage-devel/c/RGwGeJ5Tlso/m/EjRmq6gfAQAJ
.... One cause of constant confusion was that these jsmol, threejs, mathjax are served at /nbextensions. These are not proper nbextensions in Jupyter notebook ecosystem, but just static files. I suppose the proper place to serve them is /static, like /static/jsmol, ....
Fortunately we can use a Jupyter notebook option to serve /static from arbitrary directory that contains jsmol, threejs, and mathjax:
c.NotebookApp.extra_static_paths = ["/path/to/extra/static/files"]
I guess that with some changes mainly replacing "/nbextensions" to "/static", sage 3d and interact outputs all would work fine.
....
But would that work for users who want to run Sage as a remote kernel?
Replying to @mkoeppe:
But would that work for users who want to run Sage as a remote kernel?
I have no precise idea about "Sage remote kernel". I just might say that the static files(jsmol, threejs, mathjax) should be served at the right url (/static) by the Jupyter front end server.
The thing is that Jupyter has a clean mechanism to install and inspect nbextensions:
$ jupyter nbextension list
Known nbextensions:
config dir: /Users/mkoeppe/s/sage/sage-rebasing/local/etc/jupyter/nbconfig
notebook section
jupyter-js-widgets/extension enabled
- Validating: OK
It does not have such a facility for random static files.
Replying to @mkoeppe:
The thing is that Jupyter has a clean mechanism to install and inspect
nbextensions:$ jupyter nbextension list Known nbextensions: config dir: /Users/mkoeppe/s/sage/sage-rebasing/local/etc/jupyter/nbconfig notebook section jupyter-js-widgets/extension enabled - Validating: OKIt does not have such a facility for random static files.
Do you have jsmol, theejs, mathjax in the list above? I guess not. It seems that jsmol, theejs, mathjax are not (and should not be) installed as proper nbextensions, but they reside in /nbextensions folder (by the symlinks) just to be served as static files for Sage graphics.
That's right, they are not properly installed as nbextensions. My point is that they should.
Replying to @mkoeppe:
That's right, they are not properly installed as nbextensions. My point is that they should.
and my point is that they should not :-) but I am not sure...
https://github.com/jupyter-widgets/widget-ts-cookiecutter could be used for creating the package, but it has more complexity than what we need
Description changed:
---
+++
@@ -1,8 +1,9 @@
-https://groups.google.com/d/msg/sage-devel/mWTD0_iBwKc/wvhk-VtKBgAJ
+As a follow-up to #30972, we provide a pip-installable package (nbextension) that makes (all supported versions of) the `threejs-sage` offline script available to a Jupyter notebook.
-broken by #29411
+This allows users to set up a system Jupyter notebook in which Sage is fully functional (including offline 3d graphics), without having to manually create symlinks from a Sage installation to the system's nbextensions directory.
see also:
+- https://groups.google.com/d/msg/sage-devel/mWTD0_iBwKc/wvhk-VtKBgAJ
- https://groups.google.com/d/msg/sage-support/Jaiqx-KTwM0/BVxEp1gxAwAJ
- #30296 System package information and `spkg-configure` for `mathjax`This is now
https://github.com/sagemath/threejs-sage/pull/1/files
Author: Matthias Koeppe
Last 10 new commits:
5e1ca4e | src/sage/repl/ipython_kernel/install.py, src/sage/env.py, build/pkgs/sage_conf: Change from threejs to threejs-sage |
938bff4 | Read required threejs version from ext_data |
f847738 | src/sage/repl/rich_output/backend_ipython.py: Another change threejs -> threejs-sage |
d13177d | src/sage/repl/rich_output/display_manager.py: Fix up imports |
47e795b | Fix 404 fetching three.min.js from Jupyter/JupyterLab |
f74a7d8 | Merge branch 't/30972/versioned_installation_of_threejs' into t/30123/repackage_sage_s_cropped_threejs_as_a_pip_installable_package__nbextension__jupyter_threejs_sage |
11693be | build/pkgs/threejs: Switch to pip-installable package jupyter-threejs-sage |
7faf40d | build/pkgs/sage_conf: No longer define THREEJS_DIR |
d08dbf2 | sage.env: Get THREEJS_DIR from jupyter_threejs_sage |
500de1a | sage.repl.ipython_kernel.install: No longer symlink THREEJS_DIR into nbextensions |
Description changed:
---
+++
@@ -1,4 +1,7 @@
-As a follow-up to #30972, we provide a pip-installable package (nbextension) that makes (all supported versions of) the `threejs-sage` offline script available to a Jupyter notebook.
+As a follow-up to #30972, we provide a pip-installable package (nbextension) that makes (all supported versions of) the `threejs-sage` offline script available to a Jupyter notebook:
+
+- https://pypi.org/project/jupyter-threejs-sage/#description
+- https://github.com/sagemath/threejs-sage/pull/1
This allows users to set up a system Jupyter notebook in which Sage is fully functional (including offline 3d graphics), without having to manually create symlinks from a Sage installation to the system's nbextensions directory.
Branch pushed to git repo; I updated commit sha1. New commits:
74ade58 | build/pkgs/threejs/install-requires.txt: New |
Input from downstream packagers would be very welcome
Can we avoid the b1 stuff in the future? What is it even indicating.
So it puts a copy in site-package that is "importable" and a copy in share/jupyter/nbextensions. Seems to behave decently at install.
The branch otherwise look sane but I haven't fully tested it. I just focused on looking at the new packaging. If 'b1' was for after 'b', using 'c' would have been ok :)
Replying to @kiwifb:
Can we avoid the
b1stuff in the future? What is it even indicating.
That's "beta" 1 (because some metadata may need polishing). This is just the standard Python normalization of version numbers
Replying to @kiwifb:
So it puts a copy in
site-packagethat is "importable" and a copy inshare/jupyter/nbextensions.
That's right, same behavior as widgetsnbextension
Replying to @mkoeppe:
Replying to @kiwifb:
Can we avoid the
b1stuff in the future? What is it even indicating.That's "beta" 1 (because some metadata may need polishing). This is just the standard Python normalization of version numbers
Can't quite believe how that little bit just destroy the ebuild file format. Just 'b' would pass, but 'b1' with a meaning of 'beta 1', I technically should transform that into _beta1 and then retransform it inside the ebuild. If it is standard python versioning someone needs to do something Gentoo side to allow those names (other than argue that they are wrong or stupid).
Replying to @mkoeppe:
Input from downstream packagers would be very welcome
Tested with the Arch package, everything works. +1 from me.
Replying to @kiwifb:
Replying to @mkoeppe:
Replying to @kiwifb:
Can we avoid the
b1stuff in the future? What is it even indicating.That's "beta" 1 (because some metadata may need polishing). This is just the standard Python normalization of version numbers
Can't quite believe how that little bit just destroy the ebuild file format. Just 'b' would pass, but 'b1' with a meaning of 'beta 1', I technically should transform that into
_beta1and then retransform it inside the ebuild. If it is standard python versioning someone needs to do something Gentoo side to allow those names (other than argue that they are wrong or stupid).
Hopefully we can quickly proceed to an actual release so that this problem will go away.
Replying to @mkoeppe:
Replying to @kiwifb:
Replying to @mkoeppe:
Replying to @kiwifb:
Can we avoid the
b1stuff in the future? What is it even indicating.That's "beta" 1 (because some metadata may need polishing). This is just the standard Python normalization of version numbers
Can't quite believe how that little bit just destroy the ebuild file format. Just 'b' would pass, but 'b1' with a meaning of 'beta 1', I technically should transform that into
_beta1and then retransform it inside the ebuild. If it is standard python versioning someone needs to do something Gentoo side to allow those names (other than argue that they are wrong or stupid).Hopefully we can quickly proceed to an actual release so that this problem will go away.
That's just venting. I can, and did, work around that.
Changed keywords from none to sd111
Hoping we can make progress on this ticket this week - https://wiki.sagemath.org/days111
Description changed:
---
+++
@@ -10,3 +10,6 @@
- https://groups.google.com/d/msg/sage-devel/mWTD0_iBwKc/wvhk-VtKBgAJ
- https://groups.google.com/d/msg/sage-support/Jaiqx-KTwM0/BVxEp1gxAwAJ
- #30296 System package information and `spkg-configure` for `mathjax`
+
+User reports:
+- https://groups.google.com/g/sage-support/c/732Dcc7gcvI/m/XzbR_y6wBwAJplease rebase
Replying to @mkoeppe:
Also, suppose we develop a way so that users can install sagelib using
pip, instead of going through the sage distribution. Would it make sense to provide a copy of whatever parts of threejs that sagelib needs as a pip-installable Python package?
Is https://github.com/jupyter-widgets/pythreejs of any relevance in this direction?
As it turns out, it is very relevant; and it would probably be best to redo the Python packaging (jupyter-threejs-sage) as a fork of pythreejs. They already have support for JupyterLab 3
Setting a new milestone for this ticket based on a cursory review.