nb_conda_kernels and jupyter notebook extensions

[skwde]

When using extensions, dependencies (to my knowledge) have to be installed into each kernel environment separately.

A typical example would be the jupyter extension Autopep8 requries the package autopep8.
Currently I have to install autopep8 in every kernel environment.

Is there a way to only install such extension dependencies to the actual notebook enviornment and not to every kernel environment I want to use in a notebook?

[rtbs-dev]

@skwde I actually started using nb_conda_kernels specifically to avoid this isssue:

For instance, in my base or a dedicated notebook environment (uses -c conda-forge iirc):

• nb_conda_kernels
• jupytext
• jupyter_contrib_nbextensions
• rise

This is the environment that the server runs in: ~$ jupyter notebook

Then in every kernel environment, as long as I install a kernel to it (e.g. ipykernel), I will get an option in my notebooks to use the that environment directly. Since all the extensions/tools/etc are in the environment running the notebook server, I can use them at any time, in any of the kernels.

[skwde]

@tbsexton As I see it, it is not necessarily true that

all the extensions/tools/etc are in the environment running the notebook server, I can use them at any time, in any of the kernels

If I for example use the jupyter extension jupyter-autopep8 installed on the server, I still have to install autopep8 into each environment to use the extension.
That is actually the point I tried to make in the first place.

[fcollonval]

@skwde the all points of using environments is to separate installed packages sets.

⚠️ So to succeed at sharing packages between environments, you can configure your PYTHONPATH to point to directories that store shared packages. I want to insist that by doing so, you will break the isolation of environments.

[rtbs-dev]

If I for example use the jupyter extension jupyter-autopep8 installed on the server, I still have to install autopep8 into each environment to use the extension.

UPDATE

@skwde I misread your reply. The autopep8 package is separate from the jupyter-autopep8 extension, and is kernel-specific, so indeed it does seem to need to be added to the existing kernel. I treat this as the same thing to needing ipykernel (or whatever kernel you want for the environment) in each environment. One way to manage this is to make use of env.yml files, which I have for every environment. If you want to use "DRY" principle for this, you do a few things:

• conda-env-builder, though I haven't tested it myself, seems to accomplish what you are looking for
• Use reqs.txt files, which would let you have one "central" reqs file in, say, your home dir or something, and pass it with an environment-specific one in the local folder. Note that this will not work for env.yaml files, only reqs.txt for reasons discussed here
• My solution is to just be explicit rather than implicit. Use env.yml files and type out the autopep8 dependency in a - pip: section for the environments where you expect the extension to work. This will maximize the interoperability of your environment and workflow with others, in case you end up needing to share. :)

ORIGINAL COMMENT:

Not sure if there is some other issue you're experiencing, but given the setup I described (jupyter_contrib_nbextensions installed to the server, which includes jupyter-autopep8), then all you have to do is click the "nbextensions" tab in your notebook home/tree page and check autopep8 in the list.

All of your notebooks will be able to use the extension, regardless of their active kernel. You can even use the nb_conda_kernels dropdown menu to hot-swap active kernels, and the set of available extensions will not change.

This must be the case, since the extensions only interface with the jupyter notebook server, which is running in an environment isolated from the kernels. Changing the "active kernel" will not impact which environment the jupyter server is running in, nor the set of extensions it checks for.

[skwde]

@tbsexton I agree that environment.yml are great to keep track of environments and also share them.

However I still don't think that packages, installed because they are jupyter extension dependencies such as autopep8, should be in the ipykernel environment. They should stay with the extension.
Otherwise it simply leads to duplicated installes.

Also your point regarding interoperability does not realy hold in my opinion. Just image the person I am sharing the enviornment with doesn't use the same jupyter extensions. Worse still, if the person doesn't even use jupyter than he gets an environment with not only ipykernel as a useless package but also numerous others.

[fcollonval]

The discussion is getting quite of topic.

To sum-up, the requirement of autopep8 on each environment is a consequence of the implementation in jupyter-autopep8. @skwde you could open a request on that repository to use the library installed in the server environment rather than in the kernel environment.

If you need to share packages between environment, I suggest (but this can be dangerous) to make use of PYTHONPATH.

A small comment, I don't know the rational behind the need of autopep8 requirement to be in the kernel environment. But I can image it has to do with supporting various versions of Python.