Export works with Jupyter Notebook but not JupyterLab

nbautoexport version: 0.3.1+7.gf539250
Python version: 3.6.13
Operating System: Ubuntu 18.04.5 LTS on AWS EC2 Instance

Description

I newly installed and configured nbautoexport. New/modified jupyter notebooks in the configured directory are not getting exported as scripts on save when using Jupyter Lab.

What I Did

I opened the same notebooks using Jupyter Notebook and they are exported as scripts on save. Interestingly, when I then go back to using Jupyter Lab, things are working as they should (new/modified notebooks are exported on save).

Interim workaround

As of now, nbautoexport does not work with jupyterlab when the version of nbclassic is 4.0 or greater. To get both jupyterlab and jupyter notebook to work, run pip install nbclassic=0.3.7 (earlier 0.3 versions also work)

Long term fix

Write out to ~/.jupyter/jupyter_server_config.py in addition to ~/.jupyter/jupyter_notebook_config.py. After running cp ~/.jupyter/jupyter_notebook_config.py ~/.jupyter/jupyter_server_config.py, both jupyter lab and jupyter notebook export scripts correctly regardless of the version of nbclassic

Related issue in nbclassic: link

In case anyone else finds there way here, it could be worth confirming that the notebook metadata contains the following fields under language_info in addition to anything else already there:

"metadata": {
  "language_info": {
   "file_extension": ".py",
   "nbconvert_exporter": "python"
  }

I found ensuring these elements were in the metadata helped nbautoexport to work as I expected - one of my environments wasn't creating these automatically.

@mjschlauch FYI I added some takeaways from a debug session to the issue description. The culprit was a new release of nbclassic, and the quick workaround is installing a version of nbclassic < 0.4

@klwetstone's additional notes helped point me in a direction of research on the very confusing Jupyter ecosystem. I will try to summarize some key ideas here:

There are actually effectively three UI applications in the Jupyter ecosystem that let people run and interact with "Jupyter notebooks" (i.e., ipynb files).

Jupyter Notebook—this is the original legacy application that predated JupyterLab
JupyterLab
Jupyter nbclassic—this is the original Jupyter Notebook UI but running on top of JupyterLab

One reason this is confusing is because there's been big decisions about the technology stack behind Jupyter applications that have been made over the years, that ended up getting reversed. The key ideas that I understand are:

Jupyter Notebook existed first
JupyterLab is a more general purpose IDE that reached "ready for daily use" status in 2018
In mid 2020, Jupyter Notebook got a notice that it was barely being maintained and recommended that people migrate to JupyterLab. The idea was that JupyterLab originally shared backend technology with Jupyter Notebook, but the upcoming JupyterLab 3.0 was going to migrate to new backend technology.
- Around this time, nbclassic was created as an extension on top of JupyterLab but running the Jupyter Notebook UI
At the end of 2020, JupyterLab 3.0 came out, which migrated away from the backend server used by Jupyter Notebook to the new jupyter-server backend.
In late 2021, they changed their minds and decided to keep Jupyter Notebook alive, and migrate the backend to jupyter-server to share the same backend technology as JupyterLab. This is what is known as "Jupyter Notebook v7" and is not yet released today.
Keeping Jupyter Notebook around made the reason for existence of nbclassic kind of confusing. It sounds like from here that nbclassic will be the maintained version of Jupyter Notebook v6 going forward.

It sounds like all three of these are expected to be normal entrypoints to Jupyter notebooks going forward, which means we should make an effort to explicitly support all of them.

With that in mind, it seems like here is why stuff hasn't been working.

Our code is written in a way (write to jupyter_notebook_config.py) that is the way things worked for Jupyter Notebook. This still works for Jupyter Notebook (currently v6, and earlier).

According to the jupyter-server docs, this should be instead jupyter_server_config.py for any app that uses jupyter-server as a backend. It seems like, though not totally certain, that some part of JupyterLab v2 -> v3 using jupyter-server is related to nbautoexport not working anymore, which would be explained by this fact.

Some additional complication though is that nbclassic has the functionality that it works with both jupyter_notebook_config.py and jupyter_server_config.py. Apparently, that functionality broke recently, which is the issue that @klwetstone found. But it sounds like the intent is that it will continue to support jupyter_notebook_config.py? There's also some complication that nbclassic contains some shim-like behavior that adapts JupyterLab to work with old Jupyter Notebook interfaces.

It looks like eventually even Jupyter Notebook v7 will migrate to jupyter_server_config.py, based on the documentation for it.

Here's a StackOverflow post that also covers some of the mechanics about the different backend servers.

Anyways

So I think the fix is definitely to start writing the nbautoexport code to jupyter_server_config.py.

There's a question about how we want to support current and older versions of Jupyter Notebook, which still only read jupyter_notebook_config.py and not jupyter_server_config.py (I just tested this). Jupyter Notebook v7 is in pre-release but is not out yet.

Should we write out to both files on nbautoexport install?
- This will work automagically for all users, but also everyone will end up with two files, which will be messy and potentially confusing for future debugging.
Should we add a flag to write out to jupyter_notebook_config.py only when the flag is used when running nbautoexport install?
- This is more explicit, less messy, and makes future debugging easier, but adds some friction for Jupyter Notebook users to get started.

Fascinating read.

Personally, I'd vote for 1, since I use classic notebooks pretty often. May be worth it to switch this line to an info and print the path to the config as well for debugging ease:

nbautoexport/nbautoexport/jupyter_config.py

Line 23 in eb5937d

    
           logger.debug(f"nbautoexport | Installed version is {nbautoexport.__version__}.")

Looks like the PR to fix the nbclassic issue got merged in (jupyter/nbclassic#137) which, if I'm understanding thing correctly, fixes the bug that @klwetstone originally found.