Notebook template format and example notebooks
Opened this issue · 29 comments
The jupyter-notebook repository comes with example notebooks, and the same is true for most nbextensions, like ipywidgets.
- create a new file extension for notebook templates, that cannot be modified in-place. (When opening them, we would get a copy of it).
- have a
install_example_notebookshelper function (likeinstall_nbextension) that puts the example notebooks (templates) of a given package into a dedicated directory, and make the templates always be accessible from the dashboard regardless of the current working directory.
This would be very useful in the classroom.
+1 This would be great for those us of who use Jupyter for education and teaching.
What do you think the right place for the templates directory should be. ~/.jupyter/templates?
If template there is, then it should not be a directory, but on a per file basis.
The question is if template file there is, we probably want to fork the file at opening time, but then where to put the new one ?
I'm ok on API to do that, we can bikeshed on when to trigger it. But I think it's basically a "make a copy" dialog that should get enhanced with name and location of new copy.
To clarify, my proposal was, on the one hand to have the handler for the new template file extension to behave in the following way: create a copy in the current working directory. So it would be a per-file basis as you proposed.
On the other hand, the second part of the proposal is to create a means (like a new tab in the current dashboard) to access installed templates, independently of the current working directory.
The extension could be name.tpl.ipynb so that older versions of ipython, and GitHub would still render and open templates it as regular notebooks files.
Besides it would be a very simple way to get people started on https://try.jupyter.org/ and all.
Ah, my though was that any file on the system could be template. Not on a specific dir. That the case of debian docs in /usr/share/docs/.
You actually don't really need an extension you can just assume that any read-only file is a template.
I think this goes under the much broader area of documentation. Really, any python, R, Julia project might want to ship notebooks that have documentation, examples, etc.
@SylvainCorlay We need something along these lines at BNL to distribute templates to visiting users. I implemented nbexamples this weekend. Unfortunately I only found this GH issue after the fact; but fortunately, it is very much along the lines of your proposal, especially:
The second part of the proposal is to create a means (like a new tab in the current dashboard) to access installed templates, independently of the current working directory.
Take a look. The code is rough, but I'd be interested in anyone's thoughts on the approach.
I would love this feature. I want to use it because I use Jupyter notebooks to create experimental reports. The experiment is the same, and at the end of the experiment I have the same format of report. I use the notebook to create plots, charts, etc. specific to that experiment, and then export as an html document.
Right now I have a notebook that serves as a template, which I copy each time. It would be more convenient if I could simply select File -> New -> [Some template I created].
The other major use case in my workflow is to be able to create new notebooks with my typical imports already included. Right now I have a script I execute using a %run statement to set up my environment, but then I can't share the notebook without sharing the script. The alternative is to keep other notebooks open so I can paste in the import/setup code every time I start a new notebook.
For example, whenever I do machine learning stuff, I have the same setup code (import scipy, sklearn, etc.) It would be awesome to make a notebook called MachineLearning.ipynb with all the setup code, save that notebook as a template (i.e. File -> Save -> Save as Template). Then I can select File -> New -> MachineLearning and I get a new notebook in my working directory that is a copy of the template.
In such a design, I think it makes sense to have a designated location for notebook templates (call it notebook_templates to avoid confusion with Jinja templates, etc.). Any .ipynb in this directory would show up as an option in the New menu. Selecting a template from the New menu would create a copy of that template in the working directory, just as selecting a kernel creates an empty notebook in the working directory.
The Save menu would also include an option for "Save as template", which would copy the notebook (or move it from the working directory) to the notebook_templates directory. Perhaps nbconvert (or other programatic tool) could also have an options for converting an ipynb to a template (i.e. putting a copy in the directory).
My use case is similar to Gordon.
I want to be able create a notebook (PySpark/Python in my case) from a template with all the import statements. Currently, I have to copy and paste them into new notebooks.
from pyspark import SparkContext, SparkConf
from pyspark.sql import SQLContext, HiveContext
import py4j
conf = SparkConf().setAppName("Example").setMaster("local[1]")
sc = SparkContext(conf=conf)
try:
# Try to access HiveConf, it will raise exception if Hive is not added
sc._jvm.org.apache.hadoop.hive.conf.HiveConf()
sqlContext = HiveContext(sc)
except py4j.protocol.Py4JError:
sqlContext = SQLContext(sc)
except TypeError:
sqlContext = SQLContext(sc)
More enthusiasm for this feature on Stack Overflow:
https://stackoverflow.com/questions/34002326/default-template-for-ipython-notebook-using-jupyter
I've been working on a very similar use case for a text analysis tool - the InPhO Topic Explorer. I did the following:
- Created a directory called
ipynbin my source which contains a default tutorial notebook without output. It also containscorpus.tmpl.pywhich contains all the setup code, similar to @tsailiming's example. - The
vsm notebookcommand then copies the notebooks from theipynbfile to the working directory and populates thecorpus.tmpl.pywith values from a config file and copies it into the folder ascorpus.py. - In the sample notebooks I use
from corpus import *to get all my variables in the global namespace.
For educational use, a quick-start scaffold is a great help - we were able to get a classroom of historians and philosophers using the text analysis tools very quickly.
These templates would be a really nice feature. The approach I currently use (cp path/to/my/template.ipynb path/to/my/analysis.ipynb) works well enough, but being able to choose New from Template would be better. Also having a dedicated spot those these (~/.jupyter/templates, for example) would be nice.
I don't think we need a distinct format or file extension for templates: the main thing is to provide ways to open a copy of a notebook that will be saved to a new file.
As Brian suggested, this concept goes beyond the standard notion of a 'template' because it can also be used to provide examples in documentation, so that when you open an example you get a fresh, editable and savable copy. This will likely use the same underlying mechanism as 'templates' but with a different UI.
For the template UI, would a practical approach be to save notebooks in a special folder (e.g. ~/.jupyter/templates)? Then I imagine these would be presented in a separate section of the New menu.
I was looking for a feature like this all the time. Since I commonly use notebooks in two different ways, either for data analysis or module debugging, I also need different setups for each task. Thus, some kind of template system would naturally support my workflow.
I don't think we need a distinct format or file extension for templates: the main thing is to provide ways to open a copy of a notebook that will be saved to a new file.
Agreed. Besides ipynb is rendered by github and we probably want example notebooks to be rendered!
For the template UI, would a practical approach be to save notebooks in a special folder (e.g. ~/.jupyter/templates)? Then I imagine these would be presented in a separate section of the New menu.
👍 on the subdirectory of jupyter. In doing so we would have a possible path under sys-prefix... which would make example notebook installable as part of a wheel.
Regarding the extension, we could still encode something into the filename, like foobar.tmpl.ipynb so that package authors can avoid having notebook files always modified in their repository...
@takluyver how do you think we could distinguish the different paths in case of conflict (user, sys-prefix, global...)
Should they be overloaded in the usual order?
Regarding the extension, we could still encode something into the filename, like foobar.tmpl.ipynb so that package authors can avoid having notebook files always modified in their repository...
Hmm, maybe, but they also want to be able to update the examples easily. I'd think more along the lines of using something like entrypoints to discover 'example' notebooks from installed packages. That needs a bit more thought, though, because it shouldn't be Python specific.
👍 on the subdirectory of jupyter. In doing so we would have a possible path under sys-prefix... which would make example notebook installable as part of a wheel.
I'm thinking of the ~/.jupyter/templates directory being for notebooks the user thinks of as templates, distinct from 'example notebooks'. I imagine these will be relatively few, and I don't want to make it too easy for packages to install them - especially before we've worked out a proper system for example notebooks. E.g. 'templates' could be an additional section of the 'New' menu in the notebook interface - but this only works if there aren't too many.
Example notebooks could be provided using the standard jupyter data files search path - so the precedence goes: explicit overrides, user, environment, systemwide. However, we'd need to think about how to organise them within those directories, and how to present them from the UI.
Example notebooks could be provided using the standard jupyter data files search path - so the precedence goes: explicit overrides, user, environment, systemwide. However, we'd need to think about how to organise them within those directories, and how to present them from the UI.
Yes that was my question.
E.g. 'templates' could be an additional section of the 'New' menu in the notebook interface - but this only works if there aren't too many.
When I think of my use-case for notebook templates, I think of them as essentially kernels that come with some notebook data pre-populated. This concept lends itself to including templates in the "New" menu.
Perhaps some heavy users will have so many kernels and templates that the New menu becomes unwieldy, but my intuition at this point says that this should be the exception, not the rule.
We also need to consider how to create templates. At first pass, it is probably fine to require the user (or package installer) to copy an .ipynb to the templates folder. Eventually, it may be nice to allow users to "Save as template", making it easy for someone to save the boiler-plate notebook code unique to their workflow.
We at Eniram have a couple dozen people collaborating on notebooks on our JupyterHub based server. We would benefit from this feature as well.
We actually have two distinct use cases:
- every newly created notebook should have one Markdown cell with headings for metadata we use by convention (creator, description, instructions, history) – this is what I'd call a "skeleton"
- "template" report notebooks with pre-defined computations and plots – intended to be cloned, modified (filling in e.g. time intervals and data sources) and executed by users
One hack we've thought about would be to include in "template" notebooks an "open as a copy" widget with accompanying Python code. There would be a text field for the destination directory and notebook file name, and it would be pre-populated with a sane default (e.g. the user's home notebook directory and the current date as a prefix for the notebook name). Clicking on the submit button would simply run Python code which makes a copy of the notebook in the filesystem and renders an HTML link to the new notebook.
I imagine the use cases for a template/skeleton notebook are so varied that a one-size-fits-all solution is difficult to come up with. Maybe providing the necessary building blocks, hooks and examples is a better approach?
I think @akaihola's use case could be accomplished via a new-from-template approach. The "skeleton" example is just a notebook with a markdown cell - they could call that template "Eniram Notebook", i.e. New -> Eniram Notebook. While this doesn't force ALL new notebooks to have the markdown header, it does provide a straightforward way to get the desired behavior. Use-case # 2 are just notebook templates with more content.
If the "New" menu could support nested menus that reflect the file system, then organizing large numbers of templates would not be difficult (addressing one of my thoughts from earlier).
I do agree with @akaihola that a good start would be providing the building blocks, etc.
@ellisonbg @minrk @SylvainCorlay What are your current thoughts on this?
@gnestor Looks like there is pretty strong support for the notebook template scenario and usage case. I myself (N=1) would also use it at times.
I would consider expanding the idea it beyond notebooks to all file types (why not an Markdown file). I don't really have opinions on how the feature gets implemented. Given the strong interest and the potential for this to span multiple repos (server, frontends) would it make sense to begin an enhancement proposal for it?
@timkpaine This is fantastic!! Your implementation is in line with what I was imagining. It would be nice to be able to configure the template directory(ies) from the JupyterLab settings editor (to make it accessible to users who are not comfortable in the terminal or who don't have ssh access to the server). @afshin Can a server extension access the JupyterLab settings?
@ellisonbg Care to try this out?
@gnestor agreed, sounds useful. speaking with @jasongrout on a different thread about how to save templates on the fly