Usage from Shadow-CLJS?

Question

Usage from Shadow-CLJS?

teodorlu opened this issue 6 years ago · 32 comments

teodorlu commented 6 years ago

Hello!

I'm hitting some problems trying to use Oz in a Shadow-CLJS project, and I've got a question or two.

Questions

Are anyone using Oz with Shadow-CLJS?
Should this be solved in user projects, or is it feasible and reasonable to do something in Oz to make using Oz from Shadow-CLJS a smooth experience?

Problem description

When I added (:require [oz.core :as oz]) to my ns form in a .cljs file, I got this error message:

The required namespace "cljsjs.vega" is not available, it was required by "oz/core.cljs".
The namespace was provided via :foreign-libs which is not supported.
Please refer to https://shadow-cljs.github.io/docs/UsersGuide.html#cljsjs for more information.

From what I can tell from the linked section from the Shadow-CLJS user guide,

Shadow-CLJS doesn't support CLJSJS
Oz uses Vega and Vega-lite from CLJSJS
Oz needs to keep working without NPM to give that smooth Clojure experience being able to spin up a visualization from the REPL.

I don't understand all the implications of the section in the guide. I guess it might be possible to install Vega and Vega-lite directly with NPM in my project and point Oz to those.

Teodor

Answer 1 · 2019-05-02T17:17:29.000Z

Hi @teodorlu. Thanks for raising this issue.

I would love to be able to be able to support Shadown-CLJS if possible. However, I'm a bit surprised and dismayed that it does not support CLJSJS. I know very little about Shadow-CLJS, aside from the basics, so I'm not really the person to work on this. But if someone comes up with a good way of supporting it, I'd be all ears.

Does anyone know why CLJSJS isn't supported in Shadow-CLJS?

Thanks again for posting

Answer 2 · 2019-05-02T17:47:10.000Z

Glad to hear you'd like to support Shadow-CLJS! I think his point is that Shadow can use NPM directly, thus doesn't need to go through Clojars. There's some discussion on this in the guide. For the record, this prompted me to swap out Shadow-CLJS for Figwheel, which I got working.

From what I can understand in the Shadow CLJS guide, the user might be able to fix this by providing namespaces for cljsjs.vega (and the rest), that map these over to NPM packages, which can be installed from NPM with Shadow. Thomas Heller seems to have done this for quite a few libraries already.

A possible path to a fix could be to create the required mappings, try to get them merged into the shadow-cljsjs repo, and add a section on usage from Shadow-CLJS in the README.

Answer 3 · 2019-05-02T19:23:31.000Z

Great! Glad that you have things working with Figwheel.

Looking over some of those library mappings, they seem rather straightforward. Your suggestion seems like a great path forward towards providing Shadow-CLJS support.

Thanks again!

Answer 4 · 2019-05-03T08:02:55.000Z

No problem! Glad to be able to contribute.

And thanks a lot for Oz -- I think you're really getting how Vega interaction from Clojure(Script) should be.

Being able to spin up a visualization from a Clojure (JVM) REPL is a sweet experience
... and then being able to re-use that visualization code if I'm going to create a some sort of dashboard is nice.

Keep up the good work!

Teodor

Answer 5 · 2019-05-08T15:08:04.000Z

Hi :)
I've just started to play a bit with oz, after I've watched this awesome video
I created this repo to share with you how I managed to have a proper setup with oz + reagent + shadow-cljs, as I found myself in the same situation of @teodorlu...
Right now all works smooth, the only warning remained is the following

 TypeError: "vegaTooltip.vegaLite is not a function" render_vega_lite core.cljs:35

but I can live with that for the stage of my current experiments :)
Hope it can be usefull, and keep up the good work 👍

Answer 6 · 2019-05-08T18:07:54.000Z

@ivanminutillo That sounds fantastic! I'll check it out when I get the chance.

Do you have an opinion on how this should be presented to new users?

Edit: downloaded and took it for a test run. Looks good to me, thanks for the effort!

Confirmed working with globally installed shadow-cljs from NPM
Confirmed working with editor-integrated CIDER REPL.

No issues! I think it sounds reasonable to create a PR to thheller/shadow-cljsjs and create a "Usage from Shadow-CLJS README section.

Answer 7 · 2019-05-08T20:20:45.000Z

No issues! I think it sounds reasonable to create a PR to thheller/shadow-cljsjs and create a "Usage from Shadow-CLJS README section.

I think it is a good way to go 👍

Answer 8 · 2019-05-09T09:09:40.000Z

Another question, if you don't mind.

In your shadow-cljs.edn, you include your dependencies without :exclusions for the cljsjs libs. Does that mean they still get loaded?

;; from shadow-cljs.edn
 :dependencies [[binaryage/devtools "0.9.10"]
                [proto-repl "0.3.1"]
                [metasoarous/oz "1.6.0-alpha2"]
                [reagent "0.8.1"]]

From the reagent readme, they show how to provide "your own react version":

;; ...
                [reagent "0.x.x" :exclusions [cljsjs/react cljsjs/react-dom]]
;; ...

I'm asking because I really don't know. Might be able to fix that if I throw some time on it.

Answer 9 · 2019-05-11T07:59:55.000Z

Hey, sorry for late answer... I used this template to get started and as you can see there is no trace of :exclusions in the shadow-cljs.edn file.
I am interested in what you come up with this, if you have time to investigate (i am still a newbe in the clojurescript realm :))

Answer 10 · 2019-05-14T08:13:42.000Z

No problem, open source takes time. I think my hypothesis above is right, checking now.

Answer 11 · 2019-05-14T08:26:38.000Z

@metasoarous I'm starting to feel that Oz in reality covers two use cases:

Plain Clojure data -> Vega visualizations
A system for making this actually feasible when working on a Clojure (JVM).

I'm going to assume that (1.) mostly is a pure data transformation. Please correct me if that isn't the case.

Splitting (1.) and (2.) into separate packages would allow using (1.) without pulling in dependencies from (2.) Do you think there's any feasibility to this? Vega and React would still have to be provided. To be clear, I'm not proposing we (?) start work on this, but I'd like to know your opinion.

Use case I'm thinking from: As a user, I'd like to use Oz from a normal Clojure REPL interactively, and re-use those visualizations on the web.

Answer 12 · 2019-05-14T10:21:54.000Z

@ivanminutillo, I've got some results, and a question.

CLJSJS dependency exclusion

I've verified your example project pulls down CLJSJS deps,
With some trial and error, I managed to exclude them all,
And the project still works!

See my changes for details.

shadow-cljsjs mappings

I'll create a PR to shadow-cljsjs. @ivanminutillo, this is your work, so if you'd rather have your own name in the shadow-cljsjs repo, I can delete the PR.

Edit: Created the pull request.

Answer 13 · 2019-05-15T09:11:47.000Z

great work @teodorlu ! it works like a charm with :exclusions!
no problem at all with names on pull-request 👍
I'll start to test some more data-viz with reagent and oz during the next weeks, if it interests you, we could share some experiments on a shared repository to build some materials for future examples/guides ?

Answer 14 · 2019-05-15T11:13:06.000Z

Definitely interested!

I can't promise that I'll a lot of time to spare, but I think ClojureScript + Vega really makes sense for data visualization. I'm experimenting with the role of the database as well -- what does it feel like to use a data model in datascript connected to visualizations in Oz? I imagine that workflow could be a nice place to work. Feel free to tag me, and we can take it from there!

Answer 15 · 2019-05-15T11:15:06.000Z

As for this thread, this is my plan:

Wait for the changes to shadow-cljsjs to get released
Remove the cljsjs.{vega,vega-lite,...} namespaces from the example and validate things working
Create a docs/usage-from-shadow-cljs.md or something else aligned with what makes sense for @metasoarous.

Answer 16 · 2019-05-20T12:05:23.000Z

Update: as of Shadow-CLJS 2.8.37 (pushed a few days ago), CLJSJS mappings for Vega and Vega-lite are provided upstream. I've updated the example project to reflect the changes. Proposal for usage instructions suitable for documentation follows below.

@metasoarous feel free to copy the text below where you find it suitable in the Oz docs, and edit as you will. Thanks for the discussion with you and @ivanminutillo! With this, I'm closing the issue.

Using Oz from Shadow-CLJSJS

It is possible to use Oz with Shadow-CLJSJS, but care must be taken that the right Vega and Vega-lite dependencies are provided to Oz. The Oz project depends on Vega and Vega Lite as packaged in CLJSJS. This enables Oz to be able to spin up a web server with Vega loaded from a normal Clojure application. Shadow-CLJS does not support CLJSJS, and requires JavaScript dependencies to be loadable with NPM. To use Oz from Shadow-CLJS,

Ensure that you have Shadow-CLJS version 2.8.37 or later installed. Shadow-CLJSJS before version 2.8.37 did not include CLJSJS shims for Vega and Vega lite.
Check what versions of Vega, Vega-Lite and Vega-Embed your Oz version requires by reading the Oz package CLJSJS dependencies on Clojars
Install the required versions of Vega, Vega-Lite and Vega-Embed from NPM.

Optionally, check out this example project.

Answer 17 · 2019-05-22T07:12:00.000Z

@teodorlu @ivanminutillo Epic! Thank you both so much for hashing things out with each other! Love seeing the open source juices flowing :-)

I'll put these docs up on the README for now I think. May do some more detailed docs organization eventually, but while it can all fit on one page, may as well. (FWIW, the doc dir is currently a bunch of images, output from the examples; Should move these images to the examples dir when it's time to really clean up the docs).

Splitting (1.) and (2.) into separate packages ...

I don't think I'm inclined to do this, at least for now. There's some benefit to having everything in one repo while things are still actively developing. I've burned myself on breaking things into too many subprojects before. And it sounds like using exclusions has solved the issues which came up above. Could perhaps use a note in the docs though.

I'm experimenting with the role of the database as well -- what does it feel like to use a data model in datascript connected to visualizations in Oz?

I've been thinking about this! We've been more or less extending markdown with Vega-Lite & Vega, so why not also extend Vega-Lite & Vega with Datalog? Imagine datalog queries in your various :data, :datasets and :transform args? Please feel free to open an issue if you're inspired, but know it's on my mind.

Thanks again!

Answer 18 · 2019-05-22T09:43:34.000Z

Sounds good, keep up the good work!

On the splitting issue, I'm not so sure after a few nights of sleep. Checking out the source code for your reagent component, it's really minimal. Currently, users who would like just that part could copy out that part of the code.

Answer 19 · 2019-05-22T18:30:04.000Z

Thanks for the update. You're right, those components are pretty minimal and can be copied pretty easily for now. I do have some plans for extending the configurability of the plots via the vega-view api (plugging into Vega signal chans and the like), but also expect all of that to be pretty minimal. Once that's smoothed out, if there's enough substance there and demand for pulling out a separate lib, I'm open to revisiting.

Thanks again!

Answer 20 · 2020-09-24T13:26:55.000Z

Hey @teodorlu , I attempted to load your example project, but shadow-cljs is giving me an error. I wonder if this is something you have run into before? I'm not really sure what to do to get Oz working.

[:app] Build failure:
override to file that doesn't exist
{:tag :shadow.build.npm/invalid-override, :require-from #object[java.io.File 0x4d68156d "/Users/rqf595/Code/temp/reagent-shadow-oz-example/node_modules/vega-dataflow/build/vega-dataflow.js"], :require "vega-loader", :file #object[java.io.File 0x41ebcb92 "/Users/rqf595/Code/temp/reagent-shadow-oz-example/node_modules/vega-loader/build/vega-loader.node.js"], :override "./build/vega-loader.js", :override-file #object[java.io.File 0x5d156b61 "/Users/rqf595/Code/temp/reagent-shadow-oz-example/node_modules/vega-loader/build/vega-loader.js"]}
ExceptionInfo: override to file that doesn't exist
	shadow.build.npm/find-resource (npm.clj:735)
	shadow.build.npm/find-resource (npm.clj:661)
	shadow.build.resolve/find-npm-resource (resolve.clj:123)
	shadow.build.resolve/find-npm-resource (resolve.clj:94)
...

Answer 21 · 2020-09-24T14:54:16.000Z

Hey @simongray !

I don't think I've seen your error, though I haven't used Shadow-CLJS with Oz in a while.

Guess: it seems like Shadow can't find vega-dataflow.js. I'd look into whether more CLJSJS shims are required.

Answer 22 · 2020-09-24T15:43:49.000Z

@teodorlu But in your last commit to that example project you removed all of the shims since shadow-cljs should handle it, right?

Answer 23 · 2020-09-24T16:49:22.000Z

With some help from Thomas Heller, I managed to get it to compile by adding these two explicit dependencies to the package.json file:

"vega-loader": "4.3.2",
"vega-canvas": "1.2.4"

They were required by other vega packages, but the latest versions of the packages (4.3.3 and 1.2.5) are both missing a file, causing the error.

Answer 24 · 2020-09-24T18:46:52.000Z

@simongray Thanks so much for poking at this.

It looks like this example project is still using metasoarous/oz "1.6.0-alpha2", which is quite old already, and prior to Oz's switch from Lein, cljsbuild & figwheel to deps.edn and shadow. If you update to 1.6.0-alpha27, you may find that you don't have this problem any more. If you do though, it may mean something is missing from my dependencies in Oz itself, and I'm happy to try and fix that if this turns out to be the case.

Thanks again!

Answer 25 · 2020-09-24T19:52:15.000Z

@simongray Thanks so much for poking at this.

It looks like this example project is still using metasoarous/oz "1.6.0-alpha2", which is quite old already, and prior to Oz's switch from Lein, cljsbuild & figwheel to deps.edn and shadow. If you update to 1.6.0-alpha27, you may find that you don't have this problem any more. If you do though, it may mean something is missing from my dependencies in Oz itself, and I'm happy to try and fix that if this turns out to be the case.

Thanks again!

np! I think the problem occurs with the latest Oz too. The reason I went looking for an example project is because I have a Clojure newbie intern who ran into the same missing file error while trying to set up Oz with shadow-cljs.

Answer 26 · 2020-09-25T16:29:35.000Z

@simongray I can reproduce the error message you're getting. The example hasn't aged well!

...
[:app] Configuring build.
[:app] Compiling ...
[:app] Build failure:
override to file that doesn't exist
{:tag :shadow.build.npm/invalid-override, :require-from #object[java.io.File 0x4bd68097 "/home/teodorlu/dev/teodorlu/reagent-shadow-oz-example/node_modules/vega-dataflow/build/vega-dataflow.js"], :require "vega-loader", :file #object[java.io.File 0x6d02f66d "/home/teodorlu/dev/teodorlu/reagent-shadow-oz-example/node_modules/vega-loader/build/vega-loader.node.js"], :override "./build/vega-loader.js", :override-file #object[java.io.File 0x62851a71 "/home/teodorlu/dev/teodorlu/reagent-shadow-oz-example/node_modules/vega-loader/build/vega-loader.js"]}
ExceptionInfo: override to file that doesn't exist
	shadow.build.npm/find-resource (npm.clj:735)
	shadow.build.npm/find-resource (npm.clj:661)
	shadow.build.resolve/find-npm-resource (resolve.clj:123)
	shadow.build.resolve/find-npm-resource (resolve.clj:94)
	shadow.build.resolve/fn--12497 (resolve.clj:257)
	shadow.build.resolve/fn--12497 (resolve.clj:226)
...

Answer 27 · 2020-09-25T17:03:40.000Z

np! I think the problem occurs with the latest Oz too. The reason I went looking for an example project is because I have a Clojure newbie intern who ran into the same missing file error while trying to set up Oz with shadow-cljs.

I can reproduce with latest versions of Oz and Shadow-CLJS too.

@teodorlu But in your last commit to that example project you removed all of the shims since shadow-cljs should handle it, right?

Yeah, that was because we got the shims merged into Shadow-CLJS itself. The Shadow-CLJS repo then contained manual mappings that allowed Clojars-libraries (like Oz) to use NPM directly rather than CLJSJS (Javascript files hosted on clojars). Not sure if Shadow-CLJS still handles CLJSJS dependencies this way.

Edit:

Answer 28 · 2020-09-25T19:33:45.000Z

@simongray @teodorlu OK; Thank you both.

First off, Oz uses Shadow-cljs now (as of something like alpha13, IIRC), and so doesn't use any of the cljsjs shims.

Secondly, I tried adding vega-loader & vega-canvas directly to the package.json to see if I could fix this issue directly in Oz, and while I was at it, updated all of the vega libs. Unfortunately, I then ended up getting the same "override to file that doesn't exist" error that both of you observed. As a sanity check, I went back and reverted to the previous set of vega lib versions and removed vega-loader and vega-canvas, and I'm still getting the same problem. So that sucks.

I opened an issue over at thheller/shadow-cljs#790, and put the dep updates in the vega-updates branch. Please let me know if either of you know what's going on here or how I can dig myself out of the hole.

Thanks again!

Answer 29 · 2020-09-25T22:52:26.000Z

@simongray I figured out what was causing this issue. I misinterpreted your comment about about versions, and was just using the latest vega-canvas and vega-loader. I now have things building again.

I've just released metasoarous/oz {:mvn/version "1.6.0-alpha30"}. Please let me know if that resolves the problems you had using Oz with Shadow.

@simongray Did you happen to report this issue to the vega folks?

Thanks again folks!

Answer 30 · 2020-09-27T07:46:08.000Z

Glad to hear that you got it working. I'll try out your new release next week and report back.

I haven't contacted the Vega folks.

Answer 31 · 2020-09-28T08:41:51.000Z

@metasoarous Got it working. The intern is very happy now ;-)

Answer 32 · 2020-09-29T19:16:24.000Z

@simongray Wonderful! Happy interns are a good thing :-)

I'll leave an issue for the vega folks.

Thanks again folks!