Using WebPPL packages
jkhartshorne opened this issue · 25 comments
I'm trying to include some MDP/POMDP in a chapter for my fork of probmods2. And I've been banging my head against the code for a few hours, trying to figure out how agentmodels.org includes dependencies like webppl-agents.
Likely, this is because I don't really understand how github pages get compiled, what files & code are required by github pages, what is custom to probmods or agentmodels, etc. Moreover, different javascript packages are included through different methods, sometimes in the same file (ex: the chapter.html template in probmods uses a script tag -- which I understand -- and a structure called custom_js -- which I don't).
If there are any tutorials that I should look at, I'd love being pointed at them. Here's a specific question in the meantime:
I want to use code from webppl-agents in a chapter in my fork of probmods2. agentmodels contains a file called 'package.json' that contains the following:
"dependencies": {
"webppl": "github:probmods/webppl",
"webppl-agents": "github:agentmodels/webppl-agents",
"webppl-dp": "^0.1.1",
"webppl-editor": "github:probmods/webppl-editor",
"webppl-nn": "github:null-a/webppl-nn",
"webppl-timeit": "^0.3.0"
}
So it seems like maybe this is somehow involved in making webppl-agents available? But for the life of me, I can't figure out what calls 'package.json' or under what circumstances. More concerning, I noticed that much of the code from the webppl-agents package is actually included in agentmodels version of webppl.min.js. So I'm not sure that 'package.json' does anything at all.
I'd really appreciate any help in figuring out how to do this!
Alternatively, is there a way for me to include webppl packages using webppl --require? I'm assuming not, but then I've never used WebPPL outside of a browser.
I don't know much about how probmods and agentmodels are put together, but I know something about WebPPL packages, and maybe the following will be enough to get you unstuck.
The repositories called webppl-* (e.g. webppl-dp) usually contain webppl packages. These are just regular NPM packages, (optionally) augmented with extra WebPPL stuff. These packages can be used with WebPPL from the command line (using webppl --require ...) or included with the browser version at compile time. If you have access to some version of WebPPL in a browser, you can check which packages are included using this method, by dropping to the Javascript console and inspecting webppl.packages.
For example, on agentmodels.org we have:
> webppl.packages.map(p => p.name)
< ["webppl-timeit", "webppl-dp", "webppl-agents", "webppl-nn"] (4)
So, to be clear, WebPPL doesn't look at package.json to decide which packages to make available. In agentmodels, it looks as though these scripts do the work of fetching WebPPL packages (including webppl-agents) and creating a browser version that includes them. It sounds like you might need to do something similar?
That's really helpful. Do you know how the scripts get called? I can't find an explicit call. Are they automatically called during deployment?
Do you know how the scripts get called?
@jkhartshorne No, I'm afraid not. I wouldn't be surprised if they are run manually when an update to webppl.min.js is required, but I don't know that that's the case.
@null-a So ... maybe? The issue is here is the code:
`#!/bin/bash
set -e
set -x
cd node_modules/webppl
grunt bundle:../webppl-timeit:../webppl-dp:../webppl-agents
cp bundle/webppl.min.js ../../assets/js/webppl.min.js
cd ../..
echo "Success."`
Note that the things getting bundled in aren't actually in the repo. They are included as npm dependencies, so perhaps they get installed by github during deployment. But running this locally wouldn't work.
@null-a Oh, maybe you are right after all. It looks like the node_modules directly is .gitignored. So it probably is there locally.
.... in fact, reading the readme.md it is.
So new question: The probmods readme says something about running make deploy, which deploys to probmods.org rather than to github pages. Is this obsolete?
so perhaps they get installed by github during deployment
You might get this already, but running grunt bundle:.... includes all of the package contents in webppl.min.js, which is then checked into the repo. So there's no need for any WebPPL packages to be fetched when deploying to GH pages, AFAICT.
Is this obsolete?
Sorry, I don't know. I don't really know much about the specifics of probmods or agentmodels, I only know about the package system in general.
@null-a Yep. Still not sure exactly how grunt works (reading about that now), but it explains how everything got into webppl.min.js in agentmodels. Probmods does something very different for including packages, which is part of why I was confused.
it explains how everything got into webppl.min.js in agentmodels
Yes, exactly.
Probmods does something very different for including packages
I see. It looks like the version of WebPPL used on probmods is loaded from EC2, and that it doesn't include any WebPPL packages:
> webppl.packages.length
< 0
WebPPL code also has access to the global JS environment, so another way to make extra code available to WebPPL is to load it along with the page and make it accessible via a global variable. That might be one thing probmods does.
hey guys! i believe that: make-ing and deploying is no longer necessary since we switched to github pages (readme is out of date); and, external js libraries get loaded by the webpage and webppl then just uses it from the global js environment. eg look at the .md for chapter 2 or 3, where the physics engine and some helpers are explicitly added in the header....
i'm not sure whether agent-mods works the same or whether the helpers there are compatible with probmods (though probably yes in both cases?). @stuhlmueller will know the answers to all of these questions.
@ngoodman The method you use for adding box2d to a chapter works if you are loading a single .js file. It's a pain if you need to add a large number of file, and I'm not sure if it even works for .wppl files (I can imagine reasons it would or wouldn't).
I found a pretty good tutorial on github pages, and things are starting to make a bit more sense. In particular, I didn't know what Jekyll was or how it worked, and suddenly things are getting clearer.
I'm thinking of cleaning some things up in the following way:
- Use grunt to compile and minify webppl & webppl packages & store in assets
- Use the Jekyll templates to load packages as you do
- Don't use any of the AWS-served js anymore.
- External libraries like jQuery (btw - yuck!) may still need to be added manually (not during deploy), though I'm hoping they can get added through npm/bower. We'll see.
And do everything that way. Right now, probmods is a bit of a jumble of different methods for including libraries, and that makes it a pain to understand. At the same time, agentmodels' system of compiling everything into a single .js file seems like a bad idea with respect to load time. (This is why I like the Jekyll template system probmods uses to load libraries as needed.) It also risks having multiple versions of the same code loaded.
I don't see any reason to be storing things on AWS. I suspect that's left over from pre-gh days. In any case, it makes it hard to update the version of WebPPL (etc) being used. In fact, it's not even easy to find out what version is being used!
But this plan may change as I learn more, or if you tell me you really don't like those changes. (In principle, you don't have to pull them back from my fork, but I'd rather not fork permanently if possible.)
In fact, it's not even easy to find out what version is being used!
@jkhartshorne In the browser, you can do webppl.version from the JS console, and from the command line you can do webppl --version.
Yes, but you'd like to be able to tell easily just by looking at the code what versions of which package are being used. As the number of dependencies grows, this is pretty helpful. You certainly shouldn't have to fire up a server!
@null-a Dumb question, but I've been stuck on it for a while. How do I compile a webppl package into a single .js? I know how to use the grunt bundle function written for webppl to bundle a webppl package into webppl.js. But what if I don't want webppl.js? What if I only want (for instance) a .js file with webppl-timeit?
It feels like this should be doable, but unpacking the source code is taking me a while because there is a lot of it and I don't know where to look.
How do I compile a webppl package into a single .js?
That isn't something you can do at present, I'm afraid.
@null-a Are you sure? Somebody compiled webppl-viz somehow and stuck it on AWS. As far as I can tell, running grunt bundle within webppl-viz bundles the javascript but not the WebPPL code. Or am I reading that gruntfile wrong?
@jkhartshorne Sure as I can be that there's no built-in method for packing up arbitrary WebPPL packages into a single js file, suitable for browser use. Nothing stops individual packages cooking up their own scheme to do something along these lines of course.
@null-a any advice on how to cook one up? I've been looking at how to make a variant of webppl's grunt bundle that simply doesn't bundle in webppl itself, but I'm still too shaky on how the (mostly undocumented) code works.
@null-a any advice on how to cook one up?
@jkhartshorne No, and I'd be unlikely to recommend doing so. If you only care about making JS stuff available in the browser, you don't need to worry about any of this. (You can just make an extra global variable available in the page, and WebPPL programs can get at it.) If you want to make extra WebPPL code available in the browser, then I recommend bundling it into webppl.js.
If whatever you're trying to achieve doesn't fit with either of those approaches, I'm happy to discuss alternatives. (ETA: Though I stress, I'm not personally familiar with the specifics of probmods or agent-models.)
that simply doesn't bundle in webppl itself, but I'm still too shaky on how the (mostly undocumented) code works.
I think this would be trickier than you anticipate, it's certainly not simple.
Well, I didn't mean simple for me, certainly. But the alternative solution implemented by agentmodels.org -- which is to bundle all WebPPL packages into webppl.js for use in a browser -- seems far from ideal, particularly as the number of packages grows.
It's been a while since I worked on any of these web books, but Paul (null-a) is right -- by far the easiest way to deliver webppl and a bunch of packages in the browser is to bundle all the packages into webppl.js
seems far from ideal
@jkhartshorne I agree that this isn't optimal. I'd be happy to chat more about what would need to change in WebPPL to improve the particular aspects of the package system you care about, but I just wanted to give a sense of how much work that would be before you got too far into it.
I went with bundling everything. If main probmods wants to use the chapters on social cognition I've been writing, that will need to be pulled.