nicolaskruchten/jupyter_pivottablejs

Not working in jupyter within VMware vm

rickFanta opened this issue · 18 comments

rickFanta commented

Hiya Nicolas,

I'm finding that jupyter launches pivots on my laptop just fine, but am having issues at work. [On the latter, I'm behind a firewall and also running on a vmware instance in stand-alone/non-bridged mode. The proxy settings seem to be fine otherwise, and I even set HTTP_PROXY and HTTPS_PROXY environment vars for anaconda to pickup automatically as well as setting ssl_verify to False.] The pandas dataframes load just fine, but I'm not seeing anything in the cell where I try to display a pivot_ui using even the simple examples that you've provided.

I've even started jupyter with --debug and looked at the logs, but don't see what's going on yet. Any suggestions for how to debug this?

Much obliged for your indulgence on this, and all your fine efforts.
Rick

p.s. Saw your note elsewhere on the output path issues. When I set the output file path to an absolute location (e.g. "/blah/bleh/test.html") I did get a 404, but when I just specified a relative path ("test.html") it failed silently again (but wrote "test.html"). Also, I do not see the "pop out" text within the output cell.... it's perfectly blank....

p.p.s. I'm also trying to get Chrome installed on Redhat to use the Javascipt debugging extensions to better isolate the issue, but haven't been successful there either yet.

nicolaskruchten commented

Which version of Jupyter are you using? Is it the same on your laptop and at work? Is the Jupyter configuration the same or is one running under a different URL path than the other? That should help narrow down the delta...

rickFanta commented
rickFanta commented
nicolaskruchten commented

Hmm, the "http://localhost:8888/static/pivot-ajax/c3.min.css was not loaded" message suggests that there is some other weird stuff happening here, because this library only loads resources from CDNJS, not locally, so there should be nothing referring to "pivot-ajax/c3.min.css"...

rickFanta commented
rickFanta commented

Hiya.

I still haven't found a solution for this. Can you think of some things that help with the debugging?

e.g. Is there a similar JavaScript only example that could be embedded in Jupyter that I could then debug directly using browser based debugging tools?

If it helps, I'm seeing other examples work that embed d3 directly into jupyter Javascript directly (e.g. http://localhost:8889/tree/Downloads/embedded_d3_example-master). I think that that implies that it's not an issue with pulling the javascript libs that you use in your index.html template (that pull via ajax from the cdnjs.cloudflare.com repos).

Many thanks,
Rick

nicolaskruchten commented

I'm afraid I really don't know what to suggest for debugging... It really does seem like something in your local setup is interfering with dependency loading, but I'm not sure why.

There are a few moving parts to the execution of this thing:

  1. In Python we use Pandas to write a file to disk (this appears to work)
  2. In Python we use Jupyter to output an <iframe> tag that causes that file on disk to render
  3. While rendering, the browser fetches external dependencies
  4. After all dependencies are loaded, the browser's Javascript engine executes PivotTable.js

The error messages you posted above suggest that the issue is in #3 so neither Python-layer nor Javascript-layer debugging will help you.

Perhaps you can try executing the cell, and then using the Dev Tools inspector to look at the generated source of the resulting <iframe> ?

rickFanta commented

Oy vey.... Despite myself I finally got it. [It's not easy being me.... you should try it for a day... ;^) ]

It was as simple as a port conflict between Jupyter and Hue (the interface for Hadoop/big data). I think that the notebook looks for the Javascript dependencies on Port 8888 whether Jupyter is running there or not.

Btw, there are about 2 smallish Pandas changes in your excellent mtl311 package that could help. [The api for things like pivot tables has changed a bit recently. Should I contribute that to the trunk or post them to you on the issues list for that project?]

Thanks again for all your indulgence and help. It was certainly very much appreciated.

Best,
Rick

nicolaskruchten commented

Glad you worked it out!

Re the mtl311 repo, it's kind of a snapshot in time of the analysis that I wrote at the time, so I'm not sure how worth it it is to keep it up to date with the latest version of pandas... Perhaps I should add info about the version that was used :)

rickFanta commented

I think there may still an issue with the pivottable-master project ...

The Hue UI didn't like me switching it's port (in Cloudera "Hadoop"), so I had to switch the Jupyter port via --port 8800 on the command-line during startup. However, when I do that the pivot tables still fail since they pull from http://localhost:8888/static/pivot-ajax/ (i.e. all resources like d3.min.js, jquery, c3, etc all have those static URLs instead of pulling from cdnjs).

The pivottablejs.html template in the pivottable-master github project hardwires the jupyter port to 8888 (which fails if that port gets changed).

I'm looking for a fix (whereby Javascript or the python could identify the current port and change the template) and will pitch that to you for the pivottable-master project if I figure it out.

Your thoughts are welcome. Thanks again.

nicolaskruchten commented

I'm really puzzled by what your system is doing because if you look at the source, nothing is being loaded from :8888 ... I'm not sure what you mean by "The pivottablejs.html template in the pivottable-master github project hardwires the jupyter port to 8888 "

rickFanta commented

Sorry, I'm incorrect on the cause. However, I am seeing the port number be 8888 in that generated file.

To see what's happening, try running jupyter on a different port and then try a pivot table example.

nicolaskruchten commented

I'm really confused here... What generated file are you looking at? Here is the entirety of the code that generates a file: https://github.com/nicolaskruchten/jupyter_pivottablejs/blob/master/pivottablejs/__init__.py ... as you can see, there are no references to ports or 8888 in there at all.

rickFanta commented

Agreed. I see the 8888 port in files like mtl311-gh-pages/pivottablejs.html which (obviously) gets generated when you run the notebook. [e.g. see below]

Maybe that was in the old jupyter pivot module being pulled from cdnjs ?

The odd part is that the pivot table only seems to work when I leave jupyter on the default port, and fails immediately when I change the notebook port (to a non-used port).

Separately, while what you've done on this project is massively cool, I'm wondering how to jig it to work better with large big data sets.... One thought is to see if arrow could be used to pass data between pandas and the d3/javascript (w/o reserializing).

./mtl311-gh-pages/pivottablejs.html:9: <script type="text/javascript" src="http://localhost:8888/static/pivot-ajax/jquery.min.js"></script>
./mtl311-gh-pages/pivottablejs.html:10: <script type="text/javascript" src="http://localhost:8888/static/pivot-ajax/jquery-ui.min.js"></script>
./mtl311-gh-pages/pivottablejs.html:11: <script type="text/javascript" src="http://localhost:8888/static/pivot-ajax/d3.min.js"></script>
./mtl311-gh-pages/pivottablejs.html:12: <script type="text/javascript" src="http://localhost:8888/static/pivot-ajax/jquery.csv-0.71.min.js"></script>
./mtl311-gh-pages/pivottablejs.html:13: <script type="text/javascript" src="http://localhost:8888/static/pivot-ajax/c3.min.js"></script>
./mtl311-gh-pages/pivottablejs.html:15:
./mtl311-gh-pages/pivottablejs.html:16: <script type="text/javascript" src="http://localhost:8888/static/pivottable/pivot.min.js"></script>
./mtl311-gh-pages/pivottablejs.html:17: <script type="text/javascript" src="http://localhost:8888/static/pivottable/d3_renderers.min.js"></script>
./mtl311-gh-pages/pivottablejs.html:18: <script type="text/javascript" src="http://localhost:8888/static/pivottable/c3_renderers.min.js"></script>

nicolaskruchten commented

I'm still really confused, because this module has never hardcoded the port, and I know I've never used paths like /pivot-ajax/ in the past so ... are you really using the code in this repo? Or something else that behaves like it and also uses PivotTable.js? If you look in https://github.com/nicolaskruchten/mtl311 there are no hardcoded 8888 ports either.

nicolaskruchten commented

OK, this has been bugging me, so I dug a little deeper... You appear to be using this module: https://github.com/ContinuumIO/pivottablejs rather than mine which is at https://github.com/nicolaskruchten/jupyter_pivottablejs

The module you're using is specifically modified for "air-gapped systems" i.e. those not able to connect to CDNJS and it does hardcode the port, but you'll have to take it up with them rather than me for a fix :)

Sorry to bounce you around to various Github repos to get your problem solved...

rickFanta commented

Hey, thanks yet again for all the help. I'm a noob and was oblivious to some of the package management differences (conda vs pip) and where conda (Continuum) is pulling from; I thought that I was installing your module (rather than theirs) in trying to find the cause and was similarly very confused.

Much obliged.

nicolaskruchten commented

No worries. I didn't know that conda install pivottablejs installed a different package with this kind of behaviour. I've filed an issue with them.