CermakM/jupyter-datatables

Extension fails in Anaconda

ganesh3 opened this issue · 35 comments

Describe the bug
On executing the command init_datatables_mode() on kaggle kernel I get the error as mentioned.

To Reproduce
Steps to reproduce the behavior:

  1. Go to 'Kaggle' and create a 'Notebook'
  2. Install & Import the following in order with Internet access enabled:
!pip install jupyter-datatables
!jupyter nbextension install --sys-prefix --py jupyter_require
!jupyter nbextension enable jupyter-require/extension
# This Python 3 environment comes with many helpful analytics libraries installed
# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python
# For example, here's several helpful packages to load in 

import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
# Options for pandas
pd.options.display.max_columns = 50
pd.options.display.max_rows = 30
# Display all cell outputs
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = 'all'
from IPython import get_ipython
ipython = get_ipython()
# autoreload extension
if 'autoreload' not in ipython.extension_manager.loaded:
    %load_ext autoreload
%autoreload 2
import pandas_profiling as pp
from jupyter_datatables import init_datatables_mode
from scipy.stats import boxcox

!pip install fastai==0.7

import sys, fastai
print(sys.modules['fastai'])
from fastai.imports import *
from fastai.structured import *

from sklearn.ensemble import RandomForestClassifier
from IPython.display import display
from sklearn import metrics
import featuretools as ft
# Input data files are available in the "../input/" directory.
# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory
# Any results you write to the current directory are saved as output.
PATH = "../input/"
!ls {PATH}

init_datatables_mode()
  1. See error
AttributeError                            Traceback (most recent call last)
<ipython-input-17-3a17ff6fea07> in <module>
----> 1 init_datatables_mode()

/opt/conda/lib/python3.6/site-packages/jupyter_datatables/__init__.py in init_datatables_mode(options, classes)
     60     extensions = config.defaults.extensions
     61 
---> 62     require("d3", "https://d3js.org/d3.v5.min")
     63     require("d3-array", "https://d3js.org/d3-array.v2.min")
     64 

/opt/conda/lib/python3.6/site-packages/jupyter_require/core.py in __call__(self, library, path, *args, **kwargs)
     89         :param path: str, path (url) to the library without .js suffix
     90         """
---> 91         self.config({library: path}, shim=kwargs.pop('shim', {}))
     92 
     93     @property

/opt/conda/lib/python3.6/site-packages/jupyter_require/core.py in config(self, paths, shim)
    141         self._msg = {'paths': self.__LIBS, 'shim': self.__SHIM}
    142 
--> 143         self._config_comm.send(data=self._msg)
    144 
    145     def pop(self, lib: str):

AttributeError: 'NoneType' object has no attribute 'send'

Expected behavior
Should run successfully

Desktop (please complete the following information):

  • Kaggle Kernel
  • Mozilla Firefox/Chrome
  • Any version

Hello @ganesh3 and thank you for testing this out!

AFAIK Kaggle has a custom Jupyter kernel I didn't try to run it there neither do I know how much the kernel is actually different from the standard kernel.

So this remains to be explored and potentially enhanced. Thank you for the report!

Thanks @CermakM. I have tried this on a local installation of a Jupyter Notebook and it still gives the same error.

I have copy pasted the packages that once installed creates the issue when init_datatables_mode() is called.

Regards
Ganesh Bhat

Hey @ganesh3 ... well, if it happens locally as well, that's interesting. Can you open up a new issue for that and reference this one, please? Also, could you possibly share console logs? (From the Web Developer tools, usually F12 on the keyboard)

Cheers,
Marek

Hi, I'm having this problem too but not on kaggle. I've tried using it on my own jupyter installed with Anaconda and on a docker container (https://hub.docker.com/r/jupyter/datascience-notebook/)
Both give me the exact same error

@vjeronymo2 thanks for the input ... perhaps the problem is Anaconda, actually, as this seems that's what you two have in common.

Can you share the console logs here, please?

I ran the following commands on jupyter notebook cell instead of the linux prompt command now and it worked

!pip install jupyter-datatables
!jupyter nbextension install --sys-prefix --py jupyter_require
!jupyter nbextension enable jupyter-require/extension
import numpy as np
import pandas as pd
from jupyter_datatables import init_datatables_mode
init_datatables_mode()

Maybe a service was unavailable when I run these commands on the first time?
Great work though! Thanks for sharing

Interesting, I don't know why it works now and it didn't before, but I am certainly glad it does :D ... There's no service so that certainly is not the case.

Cheers,
Marek

I got another container running to reproduce the error and it failed successfully (HA!)

image

But I didn't get any useful log there
Then I refreshed the page and run this cell multiple times to get it to work
Weird...

import numpy as np
import pandas as pd
from jupyter_datatables import init_datatables_mode
init_datatables_mode()

OK, I will set aside some time to check it out in the container and see what's happening in there. Might be that some component takes a bit longer to load which causes the trouble.

Same issue here, using Anaconda, fails on init_datatables_mode() with traceback as provided by OP.

I am having troubles reproducing the issue on Anaconda. I pulled the most recent notebook image from https://hub.docker.com/r/jupyter/datascience-notebook/tags/?page=1 and I cannot reproduce it. Everything seems to work just fine.

This is may be a stupid question, but ... did you reload ("Restart & and Clear Output" and hit F5) after installing the extension? Reloading is required in order for Jupyter to register a new extension.

@CermakM Yes, I tried reloading, restarting and clearing output, and I also restarted jupyter completely, all with the same result. I am not sure how relevant this might be, but I am running

$ jupyter --version
4.4.0
$ jupyter notebook --version
5.7.4
$ python --version
Python 3.6.8 :: Anaconda custom (64-bit)

What I did find relevant however, is that I am using the SoS kernel with the Python3 module as subkernel when this error occurs. Interestingly, init_datatables_mode() works just fine when using the Python3 kernel on its own.

I will be happy to try out other scenarios should you have any suggestions/comments.

Hope this helps!

@drivenbyentropy I am sorry, unfortunately, SoS notebooks are not my field of expertise. I have no idea how they use kernels and how they communicate with the Jupyter server. Therefore, I cannot guarantee compatibility.

No worries, I just thought this might be a common theme among the reports as @ganesh3 is using the kaggle kernel and I am using the SoS kernel.

@drivenbyentropy can you give me a reproducible example so that I could at least test it out? I wasn't able to reproduce this issue yet, so that might help.

Cheers

I'm having the same issue as the above.

Jupyter local, using miniconda.

Error starting occurring after I uninstalled and reinstalled the packages. It could be an issue between using the conda env (sys env) and the user env.

I uninstalled both jupyter-require & jupyter-datatables. On reinstall, this error started occurring.

I can reproduce this error using PyCharm + a fresh pipenv environment.

Strangely, when activating the virtualenv and running the notebook outside of PyCharm, before configuring PyCharm to use the external kernel, everything works.

When managing the kernel from withing PyCharm (same virtualenv specified), the above error occurs.

Not sure if it is relevant to the problems experienced above.

Could you guys share the output of jupyter --version when this extension fails please?

Jupyter version 4.4.0

@ZachariahRosenberg the output of jupyter --version should look something like this:

jupyter core     : 4.6.1
jupyter-notebook : 6.0.1
qtconsole        : 4.4.4
ipython          : 7.9.0
ipykernel        : 5.1.3
jupyter client   : 5.3.4
jupyter lab      : not installed
nbconvert        : 5.6.1
ipywidgets       : 7.4.2
nbformat         : 4.4.0
traitlets        : 4.3.3

At least on my machine... If it's not the case on yours, can you give me the versions of these packages?

I tried to reproduce this issue in miniconda3.

Current setup:

python --version
# Python 3.7.4
jupyter --version
# jupyter core     : 4.6.1
# jupyter-notebook : 6.0.2
# qtconsole        : 4.6.0
# ipython          : 7.9.0
# ipykernel        : 5.1.3
# jupyter client   : 5.3.4
# jupyter lab      : not installed
# nbconvert        : 5.6.1
# ipywidgets       : 7.5.1
# nbformat         : 4.4.0
# traitlets        : 4.3.3

  1. Immediately after the installation, I do get the error. That's okay, Jupyter didn't get the chance to load the extension yet.
  2. Hit F5 to reload

Done. Everything works properly surprisingly enough even on Python 3.7.

@beanaroo I am sorry, but this is a Jupyter Notebook extension, it's not meant to be used within PyCharm environment.

@CermakM PyCharm has built-in support for Jupyter and Anaconda. Please see https://www.jetbrains.com/pycharm/promo/anaconda/

It works really well when starting the kernel externally, not so much when started by PyCharm. I presume this is a shortcoming on their end but thought it be worth mentioning in case the behaviour is related.

@beanaroo I am sure it has, but the extension communicates with the Jupyter Notebook frontend as well (which is, of course, different in PyCharm).

@CermakM I see the latest commit, so now the error (as you've caught) is:

CommError: Comms haven't been initialized properly.. HINT: Try reloading <F5> the window.

Reloading doesn't remove this error - all other extensions are loaded. I see in the console

Loading extension: jupyter-require/extension

And there isn't any error.

My Jupyter version just displays:

jupyter --version
4.4.0

Pulling individual packages:

Name: qtconsole
Version: 4.4.4

Name: ipykernel
Version: 5.1.0

Name: ipywidgets
Version: 7.4.2

Name: nbconvert
Version: 5.5.0

Name: nbformat
Version: 4.4.0

Name: jupyter-client
Version: 5.2.4

Name: jupyter-core
Version: 4.4.0

Name: traitlets
Version: 4.3.2

If helpful, I'm running Brave browser v 1.0.1, which is on Chromium 78.0.39

I've tried setting up a new conda env, but this had the same error. I also wonder if it has anything to do with dependency loading and Jupyter Paths. This is the output from jupyter --paths:

config:
    /Users/z/.jupyter
    /Users/z/miniconda3/envs/env1/etc/jupyter
    /usr/local/etc/jupyter
    /etc/jupyter

data:
    /Users/z/Library/Jupyter
    /Users/z/miniconda3/envs/env1/share/jupyter
    /usr/local/share/jupyter
    /usr/share/jupyter

runtime:
    /Users/z/Library/Jupyter/runtime

@ZachariahRosenberg Thanks! That is a nicely done report! :) I'll try to set up the environment accordingly when I find some spare time and see if I manage to reproduce the error.

Just one more package please: What's your jupyter-notebook version?

Also, if possible, would you also paste here the whole output of your web console with your debug mode turned on? Except for Loading extension: jupyter-require/extension there should also be quite a lot of messages with respect to the Comm initialization.

@CermakM My pleasure - when I originally had the package working, the datatables you've designed were really useful and well done. So I'm motivated to get it back!

Name: notebook
Version: 5.7.8

Here is a copy+paste of the inspector console:

Logs
Tue,
26 Nov 2019 14:40:57 GMT[ 
   requirejs
]DEBUG:Require config:{ 

}proto:Object
logger.min.js:1 Tue,
26 Nov 2019 14:40:57 GMT[
requirejs
]DEBUG:Comm targets registered.
logger.min.js:1 Tue,
26 Nov 2019 14:40:57 GMT[
requirejs
]DEBUG:(3) [
"Comm 'execute' registered.",
"Comm 'safe_execute' registered.",
"Comm 'config' registered."
]0:"Comm 'execute' registered."1:"Comm 'safe_execute' registered."2:"Comm 'config' registered."length:3__proto__:Array(0)
logger.min.js:1 Tue,
26 Nov 2019 14:40:57 GMT[
requirejs
]DEBUG:Extension loaded.
logger.min.js:1 Tue,
26 Nov 2019 14:40:57 GMT[
requirejs
]DEBUG:Comm:Comm {
target_name:"config",
comm_id:"config.JupyterRequire#1574779257.311282",
_close_callback:null,
_msg_callback:null,
kernel:Kernel
}comm_id:"config.JupyterRequire#1574779257.311282"kernel:KernelWebSocket:ƒ WebSocket()comm_manager:CommManager {
comms:{

},
targets:{

},
kernel:Kernel
}events:w.fn.init [
w…w._Events,
_original_trigger: ƒ,
trigger: ƒ
]id:"18c95f22-77aa-4830-ad4d-06fbbcfcfd3c"info_reply:{
status:"ok",
protocol_version:"5.1",
implementation:"ipython",
implementation_version:"7.5.0",
language_info:{

},

}kernel_service_url:"/api/kernels"kernel_url:"/api/kernels/18c95f22-77aa-4830-ad4d-06fbbcfcfd3c"last_msg_callbacks:{
shell:undefined,
iopub:undefined,
input:undefined,
clear_on_done:true,
shell_done:true,

}last_msg_id:"23248b8c4f5d450980f47cff4193a709"name:"python3"reconnect_limit:7session_id:"54cabcb6e64045238c350c23afae0b4e"username:"username"widget_manager:WidgetManager {
comm_target_name:"jupyter.widget",
_models:{

},
notebook:Notebook,
keyboard_manager:KeyboardManager,
comm_manager:CommManager,

}ws:WebSocket {
url:"ws://localhost:8888/api/kernels/18c95f22-77aa-4830…nnels?session_id=54cabcb6e64045238c350c23afae0b4e",
readyState:1,
bufferedAmount:0,
onopen:ƒ,
onerror:ƒ,

}ws_url:"ws://localhost:8888"_autorestart_attempt:0_display_id_to_parent_ids:{

}_iopub_handlers:{
status:ƒ,
clear_output:ƒ,
execute_input:ƒ,
stream:ƒ,
display_data:ƒ,

}_msg_callbacks:{

}_msg_callbacks_overrides:{

}_msg_queue:Promise {
:undefined
}_pending_messages:[

]reconnect_attempt:0_stopping:false__proto_:Objecttarget_name:"config"_close_callback:null_msg_callback:async ( msg ) =>{

}arguments:(...)caller:(...)length:1name:""proto:AsyncFunction[
[
FunctionLocation
]
]:core.js?v=20191126094037:435[
[
Scopes
]
]:Scopes[
3
]proto:Object initial message:{
header:{

},
msg_id:"879723f0-98116655d53ef5401118ba1b",
msg_type:"comm_open",
parent_header:{

},
metadata:{

},

}buffers:[

]channel:"iopub"content:comm_id:"config.JupyterRequire#1574779257.311282"data:{

}target_module:nulltarget_name:"config"proto:Objectheader:date:"2019-11-26T14:40:57.312057Z"msg_id:"879723f0-98116655d53ef5401118ba1b"msg_type:"comm_open"session:"af37dc8d-6fa95ffe22ccf3619cb08202"username:"zr"version:"5.3"proto:Objectmetadata:{

}msg_id:"879723f0-98116655d53ef5401118ba1b"msg_type:"comm_open"parent_header:date:"2019-11-26T14:40:57.202798Z"msg_id:"23248b8c4f5d450980f47cff4193a709"msg_type:"execute_request"session:"54cabcb6e64045238c350c23afae0b4e"username:"username"version:"5.2"proto:Object__proto__:Object
logger.min.js:1 Tue,
26 Nov 2019 14:40:57 GMT[
requirejs
]DEBUG:Comm:Comm {
target_name:"execute",
comm_id:"execute.JupyterRequire#1574779257.311282",
_close_callback:null,
_msg_callback:null,
kernel:Kernel
}comm_id:"execute.JupyterRequire#1574779257.311282"kernel:KernelWebSocket:ƒ WebSocket()comm_manager:CommManager {
comms:{

},
targets:{

},
kernel:Kernel
}events:w.fn.init [
w…w._Events,
_original_trigger: ƒ,
trigger: ƒ
]id:"18c95f22-77aa-4830-ad4d-06fbbcfcfd3c"info_reply:{
status:"ok",
protocol_version:"5.1",
implementation:"ipython",
implementation_version:"7.5.0",
language_info:{

},

}kernel_service_url:"/api/kernels"kernel_url:"/api/kernels/18c95f22-77aa-4830-ad4d-06fbbcfcfd3c"last_msg_callbacks:{
shell:undefined,
iopub:undefined,
input:undefined,
clear_on_done:true,
shell_done:true,

}last_msg_id:"23248b8c4f5d450980f47cff4193a709"name:"python3"reconnect_limit:7session_id:"54cabcb6e64045238c350c23afae0b4e"username:"username"widget_manager:WidgetManager {
comm_target_name:"jupyter.widget",
_models:{

},
notebook:Notebook,
keyboard_manager:KeyboardManager,
comm_manager:CommManager,

}ws:WebSocket {
url:"ws://localhost:8888/api/kernels/18c95f22-77aa-4830…nnels?session_id=54cabcb6e64045238c350c23afae0b4e",
readyState:1,
bufferedAmount:0,
onopen:ƒ,
onerror:ƒ,

}ws_url:"ws://localhost:8888"_autorestart_attempt:0_display_id_to_parent_ids:{

}_iopub_handlers:{
status:ƒ,
clear_output:ƒ,
execute_input:ƒ,
stream:ƒ,
display_data:ƒ,

}_msg_callbacks:{

}_msg_callbacks_overrides:{

}_msg_queue:Promise {
:undefined
}_pending_messages:[

]reconnect_attempt:0_stopping:false__proto_:Objecttarget_name:"execute"_close_callback:null_msg_callback:async ( msg ) =>{

}proto:Object initial message:{
header:{

},
msg_id:"5f56f973-6d6813f24f3f4fa403d87c4a",
msg_type:"comm_open",
parent_header:{

},
metadata:{

},

}buffers:[

]channel:"iopub"content:{
data:{

},
comm_id:"execute.JupyterRequire#1574779257.311282",
target_name:"execute",
target_module:null
}header:{
msg_id:"5f56f973-6d6813f24f3f4fa403d87c4a",
msg_type:"comm_open",
username:"zr",
session:"af37dc8d-6fa95ffe22ccf3619cb08202",
date:"2019-11-26T14:40:57.312888Z",

}metadata:{

}msg_id:"5f56f973-6d6813f24f3f4fa403d87c4a"msg_type:"comm_open"parent_header:{
msg_id:"23248b8c4f5d450980f47cff4193a709",
username:"username",
session:"54cabcb6e64045238c350c23afae0b4e",
msg_type:"execute_request",
version:"5.2",

}proto:Object
logger.min.js:1 Tue,
26 Nov 2019 14:40:57 GMT[
requirejs
]DEBUG:Comm:Comm {
target_name:"safe_execute",
comm_id:"safe_execute.JupyterRequire#1574779257.311282",
_close_callback:null,
_msg_callback:null,
kernel:Kernel
}comm_id:"safe_execute.JupyterRequire#1574779257.311282"kernel:KernelWebSocket:ƒ WebSocket()comm_manager:CommManager {
comms:{

},
targets:{

},
kernel:Kernel
}events:w.fn.init [
w…w._Events,
_original_trigger: ƒ,
trigger: ƒ
]id:"18c95f22-77aa-4830-ad4d-06fbbcfcfd3c"info_reply:{
status:"ok",
protocol_version:"5.1",
implementation:"ipython",
implementation_version:"7.5.0",
language_info:{

},

}kernel_service_url:"/api/kernels"kernel_url:"/api/kernels/18c95f22-77aa-4830-ad4d-06fbbcfcfd3c"last_msg_callbacks:{
shell:undefined,
iopub:undefined,
input:undefined,
clear_on_done:true,
shell_done:true,

}last_msg_id:"23248b8c4f5d450980f47cff4193a709"name:"python3"reconnect_limit:7session_id:"54cabcb6e64045238c350c23afae0b4e"username:"username"widget_manager:WidgetManager {
comm_target_name:"jupyter.widget",
_models:{

},
notebook:Notebook,
keyboard_manager:KeyboardManager,
comm_manager:CommManager,

}ws:WebSocket {
url:"ws://localhost:8888/api/kernels/18c95f22-77aa-4830…nnels?session_id=54cabcb6e64045238c350c23afae0b4e",
readyState:1,
bufferedAmount:0,
onopen:ƒ,
onerror:ƒ,

}ws_url:"ws://localhost:8888"_autorestart_attempt:0_display_id_to_parent_ids:{

}_iopub_handlers:{
status:ƒ,
clear_output:ƒ,
execute_input:ƒ,
stream:ƒ,
display_data:ƒ,

}_msg_callbacks:{

}_msg_callbacks_overrides:{

}_msg_queue:Promise {
:undefined
}_pending_messages:[

]reconnect_attempt:0_stopping:false__proto_:Objecttarget_name:"safe_execute"_close_callback:null_msg_callback:async ( msg ) =>{

}proto:Object initial message:{
header:{

},
msg_id:"c03bbb73-d241050129e81972308245f6",
msg_type:"comm_open",
parent_header:{

},
metadata:{

},

}buffers:[

]channel:"iopub"content:comm_id:"safe_execute.JupyterRequire#1574779257.311282"data:{

}target_module:nulltarget_name:"safe_execute"proto:Objectheader:date:"2019-11-26T14:40:57.313768Z"msg_id:"c03bbb73-d241050129e81972308245f6"msg_type:"comm_open"session:"af37dc8d-6fa95ffe22ccf3619cb08202"username:"zr"version:"5.3"proto:Objectmetadata:{

}msg_id:"c03bbb73-d241050129e81972308245f6"msg_type:"comm_open"parent_header:{
msg_id:"23248b8c4f5d450980f47cff4193a709",
username:"username",
session:"54cabcb6e64045238c350c23afae0b4e",
msg_type:"execute_request",
version:"5.2",

}proto:Object
logger.min.js:1 Tue,
26 Nov 2019 14:42:51 GMT[
requirejs
]DEBUG:Successfully frozen cell outputs.

And a screenshot (with most of the nested data closed, see above for a single layer further of nested info)

Screen Shot 2019-11-26 at 9 46 11 AM

Hope this helps!

Glad to hear that, thanks @ZachariahRosenberg! I hope that we can make it work.

So from the logs we actually can see that the Comms were successfully registered. Which means they really should be initialized... this is quite fishy.

@CermakM just taking a guess here:

Let's look at the call stack starting with the class RequireJS class method reload.

Starting with

from jupyter_require import require
require.reload()

Line 197 in core.py

if require._is_notebook:
    require._initialize_comms()

--> _initialize_comms():

self._is_initialized = False
... some codes ...
# initial configuration
self.config(paths={})

--> config():

if not require._is_initialized:
    raise CommError("Comms haven't been initialized properly.")

require._is_initialized was set to False in _initialize_comms(), which will cause config() to raise the CommError


I tried simply commenting out lines 159-160 in Jupyter-Require core.py

# if not require._is_initialized:
#     raise CommError("Comms haven't been initialized properly.")

And this seemed to fix the issue (see screenshot, woohoo).

Screen Shot 2019-11-27 at 11 05 30 AM

Interestingly enough, executing require._is_initialized afterward still returns False.

I'd imagine this likely has to do with a large mix of using self and require within the class, something I'd have to spend more time looking at in order to diagnose.

@ZachariahRosenberg that is some nice piece of investigation and you actually might be onto something. Few questions here.

  1. Aren't you by any change using autoreload? That could cause unpredictable behaviour and perhaps the existence of two requires, one that is actually loaded and the other that is not and causes the issues.
  2. Did you run require.reload() before that? Did it work afterwards?
  3. Is require initialized after running require.reload()?

Maybe a hotfix for this problem could be just initializing the comms from the jupyter_datatables side if they are not initialized. I assumed they were so I didn't think about this.

Try to check it out again with https://github.com/CermakM/jupyter-require/releases/tag/v0.5.0 released, please.

@CermakM works. Well done!

You may want to add in the documentation that when you reset Kernal, you have to refresh the page as well (as noted in the raised exception). It's also possible to factor the code to try to query/reconnect to the existing session, but a refresh isn't bad either.

Also, from a previous feature request, I see the addition of enable_datatables_mode, disable_datatables_mode, thanks! These work well. Your call, but you may want to update the readme docs with these additions, perhaps even change the part about init_datatables_mode() to just be enable_datatables_mode() since that also makes a call to initialize.

Thanks! I'm looking forward to using this more and will keep in touch if more improvements come to mind (or even start a pull request!)

@ZachariahRosenberg Thanks. All valid points! :) I'll try to update the docs soon along with the planned 0.4.0 release.

I am FINALLY closing this issue, then! :) Thanks a lot for the assistance and dedication to solve this issue, your help has been invaluable.

Cheers,
M