jupyter/notebook

7.0 Release Plan

jtpio opened this issue · 48 comments

jtpio commented

This is an issue for 7.0 release planning. It will continue to be updated as we plan and develop Jupyter Notebook v7.

From the Notebook v7 Jupyter Enhancement Proposal:

The next major release of the Jupyter Notebook application, version 7, will be based on the JupyterLab codebase, but will provide an equivalent user experience to the current (version 6) application.

Notebook v7 Project Board

Issues and PRs related to Notebook v7 are tracked in this board: https://github.com/orgs/jupyterlab/projects/2/views/1

We also use the 7.0 milestone to add any issue and PR relevant to the 7.0 release: https://github.com/jupyter/notebook/milestone/45

Relevant Links

gutow commented

May 28, 2022 - interface testing from perspective of one of the users who would like to maintain parity with the old notebook interface.
First: this is looking really good from the user interface perspective. I am still trying to get set up to port extensions and will comment on that later/elsewhere, as appropriate.

  1. The interface recovers most of the simplicity of the classic notebook. It is not identical, but it is still relatively simple.
  2. It is odd to have it start up in collaborative mode. I am not sure that should be the default. Although, I am not sure it causes a problem.
  3. Automatic saving on by default seems a little strange to me. That said, fewer things are likely to be lost. I think problems depend on the rate of checkpointing. If somebody makes a mistake it may be necessary to role back quite a bit. With only intentional saves, things are not accidentally overwritten. However, most students are conditioned to expect automatic saving. It might be worth having a menu item for checkpoint rate and number of checkpoints to keep.
  4. "Close and Shutdown notebook" does not exit the browser tab. This is very confusing.
  5. I believe the notebook metadata specifies the kernel. If a kernel is specified the notebook should open with that kernel by default if the kernel is available.
  6. There is no way to access cell metadata for editing. It looks as if this is being addressed in #6419.
  7. It would also be nice to have the trusted/untrusted indicator back.

Things are looking good.

jtpio commented

Many thanks @gutow for the feedback 👍

It is odd to have it start up in collaborative mode. I am not sure that should be the default. Although, I am not sure it causes a problem.

Probably you were trying using the Binder link in the README? It sets the collaborative flag by default so it's easier to test RTC on Binder:

notebook/jupyter_config.json

Line 3 in 16a297a

"JupyterNotebookApp": { "collaborative": true, "expose_app_in_browser": true }

Automatic saving on by default seems a little strange to me.

This is also a side-effect of having RTC enabled, and should not be the case when starting without the collaborative flag.

You can try the latest v7 pre-release with the following gist on Binder which doesn't have RTC enabled: https://gist.github.com/jtpio/0f3c67a6a1767360df0e79b52d74a5fe

"Close and Shutdown notebook" does not exit the browser tab. This is very confusing.

Mind opening an issue? Or maybe this could be part of #6397.

It would also be nice to have the trusted/untrusted indicator back.

Yes, there was a previous attempt at adding it in the former RetroLab repo: jupyterlab/retrolab#279
Which could be ported over here.

gutow commented

Many thanks @gutow for the feedback +1

You're welcome. Sorry about the slow reply. I was traveling.

It is odd to have it start up in collaborative mode. I am not sure that should be the default. Although, I am not sure it causes a problem.

Probably you were trying using the Binder link in the README? It sets the collaborative flag by default so it's easier to test RTC on Binder:

Actually, I was working with a clone of the repository locally. I am hoping to get things to the stage where I can test porting some of my widgets to the JLab within this interface.

"Close and Shutdown notebook" does not exit the browser tab. This is very confusing.

Mind opening an issue? Or maybe this could be part of #6397.

I think this belongs as part of #6397 and will add to that.

It would also be nice to have the trusted/untrusted indicator back.

Yes, there was a previous attempt at adding it in the former RetroLab repo: jupyterlab/retrolab#279 Which could be ported over here.

Please port.

frenzymadness commented

I'm interested in this from a different perspective. I maintain a lot of notebook/jupyter-related RPM packages in Fedora Linux. We had only Notebook packaged for a long time because Lab required too many dependencies and nobody was willing to invest time to package them. When I saw the effort to make Notebook 7 work on components from Lab, I started with packaging the long tree of dependencies and I'm almost done and Lab will soon be available in Fedora Linux as an RPM package.
Right now, we have Notebook version 6.4.12 and I'm thinking whether it makes sense to update it to 6.5.2. The problem is that Notebook 6.5.2 needs nbclassic package which we don't have yet and which is not needed for Notebook v 7. How long do you think it will take before the release of Notebook 7? Does it make sense to invest my time to packaging nbclassic and updating Notebook to 6.5.2 or better is to wait for Notebook 7?

jtpio commented

Many thanks @frenzymadness for your work on packaging! And for raising the question here 👍

whether it makes sense to update it to 6.5.2

Normally the 6.5.x line is only for preparing the transition to 7. And potentially getting some critical security updates. There is more information in this blog post: https://blog.jupyter.org/jupyter-notebook-6-5-0-release-candidate-4c229c6dab55

Does it make sense to invest my time to packaging nbclassic

At first glance I would say the current list of dependencies for nbclassic should overlap quite a lot with the ones of the current Notebook 6 and JupyterLab 3. Looking at the list I am however wondering whether all of them are required.

How long do you think it will take before the release of Notebook 7?

The Notebook 7 release is conditioned by the JupyterLab 4 release, which is tracked in jupyterlab/jupyterlab#9647. In the last JupyterLab dev meeting, we discussed entering a feature freeze by the end of February 2023, so a beta would follow shortly after. Once a beta of JupyterLab 4 is available there will likely be a beta for Notebook 7 as well, and so on until the final release. Hopefully all of this will happen in the coming month although there is no clear date yet.

  • 👍1
frenzymadness commented

Thank you for the quick and useful answer. Fedora 38 (currently in development) has its beta freeze on the 21st of February so my current plan is to package nbclassic and update notebook to 6.5 before the beta freeze and update notebook to 7 in Fedora 39. Notebook is in Fedora for years so it should remain stable, and therefore small steps are better for it. I'd also like to add Lab (alpha/beta release) into Fedora 38/39 as soon as possible so users can test it and provide feedback.

  • 👍1
frenzymadness commented

The current status of Jupyter components in Fedora Linux is that Fedora 38 (to be released in April) has nbclassic and notebook 6.5. The development version 39 (release planned for October) just got lab 4 alpha34 and I'm preparing the update of notebook to 7 alpha 14. Adding an alpha release as a completely new package to Fedora is fine because nobody has any expectations and it cannot really break anything. The situation is quite different with the update of the notebook to the alpha release because notebook is in Fedora for many years. I can do that in the development version of Fedora and it will be good for the Jupyter project because it should bring some more attention to the changes happening now but I have to be sure that the notebook (and lab) reaches stable releases before Fedora 39 reaches beta freeze in August. Otherwise, people upgrading from 38 to 39 would go from stable 6.5 to unstable 7 and that won't be good.

Do you think it's doable? Do you have any final deadline?

jasongrout commented

Thank you very much for updating these packages. We are really trying hard to release JupyterLab 4 before JupyterCon in May, and I think we are still on track to do so. (I'll let others speak to Notebook 7, though).

jtpio commented

Notebook 7 will likely follow the release plan of JupyterLab 4.

With a first beta shortly (couple of days, or 1-2 weeks) after JupyterLab 4 beta. And a similar timeline for rc and final.

Carreau commented

I just saw the "Read the migration plan to Notebook 7 ..." message. I thing there should be a way to "dissmiss for now" instead of "dismiss permanetly".

jtpio commented

a way to "dissmiss for now"

You mean as a way to remind the user later?

cc @RRosio @echarles

  • 👀1
Carreau commented

Sure, even if it's remove until page refresh.
Say I'm teaching demoing a tutorial and I just want to reclaim the vertical space right now for the next hour.

kiyoon commented

Hi, in my neovim plugin, I used Notebook's front-end APIs to interact with Notebook through neovim. For example, I used Jupyter.notebook. ... to set cell text and execute cells etc.

I don't see any way to use the front-end API's in Notebook 7 or Jupyter Lab. I prefer to just interact directly through the browser javascript, but is the only option writing a Jupyter Lab extension?

If you're interested, kiyoon/jupynium.nvim#74 is the summary of issues I have with supporting Notebook 7. Some of my points may be wrong.

Any guidance with migrating the plugin to support Notebook 7 will be much appreciated!

  • 👍2
gutow commented

@kiyoon I agree that it would make it easier to write user friendly extensions to Jupyter Lab if the javascript APIs were exposed. As I understand it (I did not work on the development of Jupyter Lab), most of this is a side-effect of efforts to make the code more modular and thus easier to maintain. I believe the developers then chose not to expose the javascript APIs directly to provide what I would call "security by effort level". It is still possible to inject invisible javascript that does bad things, but the hope is the effort level (coding overhead) to do so is high enough that attackers will go after other lower coding effort targets. Note this is just my understanding/interpretation. I was not involved in any of the discussions or development.

IMHO I believe this protection by effort level will decrease over time as more examples of plugins that do various things are developed. At some point there will be plugins with the correct boilerplate so that nefarious code can easily be built by repurposing an already developed plugin, with minor changes to the code. This will also provide easier ways for people to extend Jupyter lab to meet their needs.

At this point I suggest you look at the code for existing plugins. If you have not already, I would start here: https://jupyterlab.readthedocs.io/en/stable/extension/extension_dev.html.

kiyoon commented

@gutow Thanks for the input. Since my plugin is to interact front end mostly, it is much better to use a front end API than making an extension by design. It's a pity they decided to deprecate this. It seems like in the official website it is stated that the classic notebook will be supported for 2 years, so I'll probably use this. But I honestly would have preferred if Notebook 7 never existed, because it unnecessarily break things when there's Jupyter Lab exists as an alternative anyway. Or, it would have been nice to keep the classic notebook package to notebook, and introduce new notebook as a new name (like no nbclassic but have notebook7, labnotebook etc.). The URL shouldn't have changed to localhost:8888/nbclassic because now plugins need to check which notebook version is running after loading the page, and reconnect.

For example, if you only have nbclassic installed it will be the normal URL localhost:8888 but then if you have both notebook and nbclassic installed it will be localhost:8888/nbclassic. This kind of inconsistency shouldn't have existed in my opinion.

jasongrout commented

For example, I used Jupyter.notebook. ... to set cell text and execute cells etc.

Do you have a list of what you specific APIs you need? Like

  • Set the cell text to a specific value
  • Execute a given cell
  • what else?....

It would be really great to develop more examples in https://github.com/jupyterlab/extension-examples showing how to easily do these things.

jasongrout commented

I believe the developers then chose not to expose the javascript APIs directly to provide what I would call "security by effort level".

As a clarification: the extension system is more about making components modular, reusable, and easy to package/distribute/remix, with different components playing nicely with each other by providing and using each other's APIs. The Javascript APIs are mostly exposed (and of course as needs arise, more can be exposed), but the APIs are compartmentalized by component so those components can be remixed and reused independent of each other.

Security by effort level wasn't a design goal, largely because as you point out, it would just be an illusion anyway.

  • 👍1
kiyoon commented

Here is a list of things I used so far.

Jupyter.notebook.ncells()
Jupyter.notebook.cells_to_code([0])
Jupyter.notebook.cells_to_markdown([0])
Jupyter.notebook.get_cell(0).set_text(arguments[0])
Jupyter.notebook.get_cell(0).render()
Jupyter.notebook.get_cells()
Jupyter.notebook.insert_cell_below()
Jupyter.notebook.delete_cell(-1)
Jupyter.notebook.scroll_cell_percent(arguments[0], arguments[1], 0)
Jupyter.notebook.toJSON()    // this is to download notebook
Jupyter.notebook.scroll_manager.is_cell_visible(Jupyter.notebook.get_cell(arguments[0]))

Jupyter.notebook.kernel.name
Jupyter.kernelselector.kernelspecs
Jupyter.kernel_list.kernelspecs
Jupyter.kernelselector.set_kernel(arguments[0])
Jupyter.notebook.kernel.interrupt()
Jupyter.notebook.kernel.restart()
Jupyter.notebook.kernel.is_connected()
Jupyter.notebook.kernel.complete(arguments[0], arguments[1], completeCallback)
Jupyter.notebook.kernel.inspect(arguments[0], arguments[1], inspectCallback)

Jupyter.notebook.clear_output()
Jupyter.notebook.clear_cells_outputs(Jupyter.notebook.get_selected_cells_indices())
Jupyter.notebook.toggle_cells_outputs_scroll(Jupyter.notebook.get_selected_cells_indices())
Jupyter.notebook.execute_selected_cells()
Jupyter.notebook.save_checkpoint()
Jupyter.notebook.save_notebook()
Jupyter.notebook.scroll_manager.animation_speed = 0; Jupyter.notebook.scroll_manager.scroll_some(arguments[0])
Jupyter.notebook.rename(arguments[0])

Thanks for looking into this. My plugin basically enables vim to interact with Jupyter Notebook within vim through the front end APIs. So I need basically most of the controls, including kernel completion.

Also, my plugin doesn't install any notebook extensions. It just interacts with Notebook through Selenium browser javascript execution. Because the plugin is only about interacting with Notebook and not modifying any part of it, it works nicely if the front end APIs are exposed directly, through the browser.

jasongrout commented

I made an extension that starts to explore equivalents to classic Notebook apis: https://github.com/jasongrout/nbactions/blob/main/src/index.ts

As for interacting with JupyterLab through Selenium (or these days, perhaps it is better to use one of the more modern packages for interacting with the browser?), perhaps there can be a connector extension that exposes an API to Selenium. Or maybe even better, there is a commands framework in JupyterLab for actions that is used for menu items, keyboard shortcuts, etc. Perhaps Selenium could have basically a single entry point to invoke a command in JupyterLab, which then would be able to do anything you can do in JupyterLab from a menu item or keyboard shortcut.

Our testing framework Galata can control JupyterLab in various ways too, so there is precedence for controlling JupyterLab from browser testing frameworks.

kiyoon commented

@jasongrout Wow, thanks a lot for making this for me!

Regarding Selenium, I didn't know there are modern alternatives to Selenium. For a simple plugin like that Selenium met all requirements, but I may have been missing out on something. If I can guess the async functionality of my plugin can be improved for speed.
By the way I have zero experience in web development and javascript, which is why I found it hard to develop extensions.

So to enable this extension, I just pip install nbactions and it will expose the API's on Jupyter Lab and Notebook 7? I should check how it works soon.

Or maybe even better, there is a commands framework in JupyterLab for actions that is used for menu items, keyboard shortcuts, etc. Perhaps Selenium could have basically a single entry point to invoke a command in JupyterLab, which then would be able to do anything you can do in JupyterLab from a menu item or keyboard shortcut.

Any documentation on how to use the commands framework? If this solution doesn't require building an extension I'd maybe prefer this way. But I wouldn't want menus to pop up every time and the UI shouldn't glitch much. Keyboard shortcuts can also change and it's not going to be as smooth as the front end API.

Again I really appreciate the support!

ruler501 commented

Commands would still need extra boilerplate to handle reading and modifying cell contents wouldn't it?

jasongrout commented

So to enable this extension, I just pip install nbactions and it will expose the API's on Jupyter Lab and Notebook 7? I should check how it works soon.

Just to be clear - I just noted down a few of the equivalent commands (I think, I didn't test) for the notebook API commands, at least to give an idea of what those commands look like for JupyterLab. It's still quite a ways from exposing these to selenium, etc.

For other alternatives for controlling the browser, this article talks about some of them. I'm not sure if Playwright or Cypress is a better fit for you than Selenium.

Running commands won't pop up menus, etc. In JLab, the menus and keyboard shortcuts invoke commands under the hood to do the actual actions, and you can also programmatically invoke commands to do the actions. So you're not forcing the browser to pick a menu item or simulate pressing some keys - instead, you are actually calling the underlying function both of those things call in order to perform an action.

And @ruler501 - commands can take arguments, so you can pass in, for example, the text to put in a cell.

jasongrout commented

(And just to set expectations, I unfortunately probably won't have time to finish a guide for translating classic API calls to Jlab/Notebook 7, like I'd really want to :(. However, I'm happy to brainstorm, take a few minutes to prototype a few things like tonight or discuss options, and happy to answer questions where I can.)

jasongrout commented

Any documentation on how to use the commands framework?

Here are some docs about creating new commands (but may give you a sense for how the system works).

Here is a list of commands that the notebook component currently registers. I put a few examples (here and here, for example) in the nbactions plugin to see how commands can be invoked.

kiyoon commented

For other alternatives for controlling the browser, this article talks about some of them. I'm not sure if Playwright or Cypress is a better fit for you than Selenium.

Thanks for the link, it was very helpful. Maybe other packages would solve issues I had with Chrome and maybe minor bugs. I'll need to try them out sometime.

(And just to set expectations, I unfortunately probably won't have time to finish a guide for translating classic API calls to Jlab/Notebook 7, like I'd really want to :(. However, I'm happy to brainstorm, take a few minutes to prototype a few things like tonight or discuss options, and happy to answer questions where I can.)

Of course, this is already very helpful and I appreciate giving me the kickstarting point!

I put a few examples (here and here, for example) in the nbactions plugin to see how commands can be invoked.

I still don't see how commands are different from just using NotebookActions. How does it make it possible for Selenium to execute the commands, and not the NotebookActions.* functions?

Also, I had issues with building the extension. I raised an issue in the repo directly, so if you have time to check it would be much appreciated!

anki-code commented

Small note. If there are users of the python-powered xonsh shell and the xonsh kernel for jupyter among jupyter contributors it will be cool to test the kernel with new jupyter 7. Thanks!

jtpio commented

FYI the first beta for Notebook 7 is now available:

The documentation has also been improved quite significantly:

Please don't hesitate to submit PRs to improve the documentation, or open issues if you find bugs.

We are currently working on a blog post on medium and a post on Discourse to announce the beta more publicly, stay tuned.

Thanks and happy testing!

  • 🎉2
jtpio commented

FYI there are now two new milestones, so we can move issues not critical for the Notebook 7 final release:

jtpio commented

Today at the Notebook Weekly Meeting we discussed the release timeline for Notebook 7: jupyter/notebook-team-compass#21 (comment)

To have more time to discuss this topic the weekly meeting next week (Wednesday 12th April 2023) will start at 8 am Pacific. You can find the details to join the call here: https://github.com/jupyter/notebook-team-compass#weekly-team-meetings

Anyone is welcome to join the meeting so don't hesitate to drop by if you would like to!

cc @yuvipanda in case you would like to join since you commented in jupyter/notebook-team-compass#21 (comment)

jtpio commented

For awareness JupyterLab 4.0 final has been released: https://github.com/jupyterlab/jupyterlab/releases/tag/v4.0.0

And Notebook 7 has been updated to the JupyterLab 4 final packages: #6871

We'll be working on releasing Notebook 7 final in the coming weeks.

  • 🚀6
andrii-i commented

Currently there are 59 open issues is the Notebook 7 milestone link. If we want to release Notebook 7 in the next couple of weeks, we should identify “release blockers”. Should we create “release blocker” tag and review list of open Notebook 7 issues identifying release blockers during Jupyter Notebook Meeting on Wednesday (this can take up the entire meeting)?

  • 👍1
jtpio commented

Thanks @andrii-i. Yes doing a triage session sounds good. I won't be able to make it to the meeting this week but feel free to proceed. We can also do this asynchronously on the issues directly.

  • 👍1
andrii-i commented

During Notebook and JupyterLab meetings we triaged Notebook 7.0 milestone bugs and identified 4 release blockers. Please see details in the meeting notes: jupyter/notebook-team-compass#21 (comment)

jtpio commented

Thanks @andrii-i!

andrii-i commented

During today's Notebook call we discussed release strategy for Notebook 7 and what should be done before the release:

  1. Regarding WCAG 2.1 accessibility issues listed in #6800, we should consider an issue as blocking if it prevents someone currently using Notebook 6 to migrate to Notebook 7. Either because there is a big regression, or because something is missing in Notebook 7.
  2. We should look at Notebook JEP 79 and what is promised for Notebook 7 release within it, in particular "Critical extensions" section, RISE: https://jupyter.org/enhancement-proposals/79-notebook-v7/notebook-v7.html
  3. There are still 3 open issues identified as "Release blockers": https://github.com/jupyter/notebook/issues?q=is%3Aissue+is%3Aopen+milestone%3A7.0+label%3A%22tag%3ARelease+Blocker%22. Looking for assignees for these issues
andrii-i commented

Critical extensions mentioned in JEP 79:
- nbgrader is work in progress: jupyter/nbgrader#1684
- RISE needs updating to JupyterLab 4: jupyterlab-contrib/rise#29
- jupytext is work in progress as a separate jupyterlab-jupytext extension: mwouts/jupyterlab-jupytext#6
- IPython parallel is done as a standalone package ipyparallel: https://ipyparallel.readthedocs.io/en/latest/#

andrii-i commented

There are 2 remaining blocking issues in the Notebook 7 milestone. I wanted to initiate a discussion about next steps and what specifically needs to be done to prepare for the release (other than resolving blocking issues).

From my perspective, we have achieved a decent level of consistency with JEP 79 and are ready for RC but we need an alignment as a team. If there are any issues that have not been sufficiently addressed and should be tackled before the release, please share.

andrii-i commented

We've now resolved all the remaining blocking issues in the Notebook 7 milestone 🎉.

We're now preparing to release Notebook 7 Release Candidate 0 soon. If no critical issues arise after RC0, we plan to do a full release a week later.

  • 🎉6
andrii-i commented

Update on Notebook 7 blogpost: jupyter/notebook-team-compass#24 (comment)

fecet commented

In jupyter notebook v7, is there any way to list all kernels live in notebook? Say, I open 3 notebooks in jupyter notebook

../dir1/notebook1.ipynb
../dir1/notebook2.ipynb
../dir2/notebook3.ipynb

and I would like to know theire correspoding kernel connection files say

../dir1/notebook1.ipynb  python 3.8(env1)  kernel-0e32...json
../dir1/notebook2.ipynb  python 3.8(env1)  kernel-7b63...json
../dir2/notebook3.ipynb  python 3.10(env2)  kernel-53c3...json
...

like lkhphuc/jupyter-kernel.nvim#9.
I found it's diffuct to do so in notebook v6, is v7 provide any similar thing?
I'm a https://github.com/kiyoon/jupynium.nvim user and it seems that that need a lot effort to migrate to v7, as dicussion before in this issue. So I'm wondering if notebook v7 provide enough things for me to build a simple repl workflow.

jtpio commented

FYI 7.0.0r0 is released 🎉

Please try it and report issues if you find any:

Thanks!

  • 🚀3
gutow commented

Non-blocking, but issue to consider:

  • Choosing File>Close and Shutdown Notebook closes and shuts down the notebook, but it does not close the tab/window as I would expect.

My preliminary testing hasn't shown any other UX issues or unexpected incompatibilities with existing modules I use. This is looking good.

Thanks for the work on this!

jtpio commented

Thanks @gutow for testing and reporting 👍

Would you like to open an issue, so we can have a look during the RC period?

  • 👍1
andrii-i commented

I found it's diffuct to do so in notebook v6, is v7 provide any similar thing?
I'm a https://github.com/kiyoon/jupynium.nvim user and it seems that that need a lot effort to migrate to v7, as dicussion before in this issue. So I'm wondering if notebook v7 provide enough things for me to build a simple repl workflow.

@fecet thank you for bringing this up. Would you like to test if this feature is available in v7.0.0rc0 (can be installed with pip install --pre notebook)? If feature is missing, creating new issue / enhancement request to track and consolidate work around it would be a good next step

ericsnekbytes commented

In the Notebook weekly meeting, we discussed some updates to release planning:

  • Wait until @jtpio is back to release
  • Target July 19th as a tentative new Notebook 7 release date (release party during the weekly!)
  • Plan to communicate (discourse, etc) the impending release in multiple places to community, important extension authors (RISE, NBGrader)
  • We have an opportunity to work on some accessbility issues in the interim
  • Can make additional RC releases as well in the interim
  • Put blog post up after pip and conda packages are out
  • 👍3
kiyoon commented

@andrii-i For jupynium.nvim, all I need is a front-end Javascript API so I can interact with the notebook through browser scripts, without writing extensions. If this is supported, jupynium.nvim can be easily rewritten.

I created the issue #6949

jtpio commented

For reference a first nbgrader pre-release with support for Notebook 7 has just been released (thanks @brichet!): https://github.com/jupyter/nbgrader/releases/tag/v0.9.0a0

  • 🎉2
jtpio commented

Notebook 7 is released 🎉

Please report issues if you find any 🙏

We will continue working on addressing issues, improving documentation and working with the community to ease the transition.

Thanks all!

  • 🎉4