WebIDE for development

Question

WebIDE for development

FWeinb opened this issue 6 years ago · 25 comments

As discussed in #102 it would be great to have the ability to change/create/extend the script that is running on the device using some kind of WebIDE. It was suggested there that it would be possible to compile the JS code on the device itself.

I experimented with a different solution but don't know how useful it would be. I successfully compiled xsc/xsl using emscripten and am able to compile JS in the browser now. I did not look into the whole architecture of this project, my initial idea is to transfer the generated buffer to the device and then execute them there.

That is the point where I am currently stuck I don't know how I would go and execute the payload compiled by the browser.

Any pointers to how/if this could work?

Answer 1 · 2019-01-22T17:07:03.000Z

Very cool that you have xsc and xsl running in the browser. Delivering byte code to the embedded device is preferred over delivering source code for several reasons. It saves flash by avoiding the need to deploy the parser to the device, and it allows larger scripts to be downloaded (as the parser is contained by free RAM on the embedded device).

From XSL you should have a .xsb file (XS binary). That needs to be transferred to the device flash, for example using an HTTP PUT. I assume that's easy from the browser? After that, you'll need a small shell app to install and execute the binary. I can prepare that for you. It isn't very complicated, but I may not have time until the weekend.

Answer 2 · 2019-01-22T17:09:36.000Z

Yeah making a PUT/POST request isn’t the problem. Would be great if you could provide me with an example of how I would startup the VM running the *.xsb file on an ESP.

Answer 3 · 2019-01-22T17:13:40.000Z

Of course. I'll put together a small project.

Answer 4 · 2019-01-22T18:54:05.000Z

@FWeinb Do you have a pointer to your approach to compiling with emscripten? We got something similar working at https://github.com/Agoric/moddable/tree/wasm-spike to get XS to compile to wasm. We would like other eyes and usage on it, getting tests to run within that runtime, and to eventually get it into a pull request, so any info on related projects is welcome.

Answer 5 · 2019-01-22T19:32:05.000Z

@FWeinb - one question: what device target are you using? I was assuming ESP8266 but your comment didn't say.

Answer 6 · 2019-01-22T20:43:52.000Z

@phoddie ESP8266/ESP32 would be great.
@dtribble i will have a look at your work. Don’t have that much time this week, has to wait till the weekend.

Answer 7 · 2019-01-27T04:45:59.000Z

@FWeinb - I put together an example for you. It consists of a host application to run on the ESP8266 and three sample mods you can run on it. The only only tools needed are xsc, xsl, and some way to do an HTTP PUT. The details are a bit involved, so I also wrote a document explaining it. All the code and documentation is available here Building and Installing JavaScript Mods.

The ability to dynamically install modules compiled to byte code is definitely an advanced feature, one that we are just starting to use. That's why there wasn't an example or documentation already in place. If you run into problems or have questions, please let me know. I've very curious to see this working from the browser. ;)

Answer 8 · 2019-01-30T13:16:57.000Z

@phoddie Sorry for the slow feedback. I was sick during the weekend, could not look into it.

Will have a look at it this weekend.

Answer 9 · 2019-01-30T15:30:35.000Z

@phoddie
Just had some time and need your feedback, sorry for double posting here.
I got xsc to compile and it will generate two files code.xs.h and code.xs.c based on an code.js, you can test this here: Basic WebIDE

Do I really need to compile these two files using an C-Compiler to then use xsl to link it? That would be a deal breaker for me. I don't think I want to use gcc in the browser.

Would it be possible to extract xsSymbols and xsCode and send these to an ESP without having to compile/link it?

If that is the case I could write a simplified version of xsc that reads from stdin and writes to stdout without generating all these files, would reduce the js code size a lot.

Answer 10 · 2019-01-30T15:53:23.000Z

There's no need to run GCC to build JavaScript byte code binaries. The instructions posted here show the xsc and xsl options to use. The only output of xsc in this case is a .xsb file.

Answer 11 · 2019-01-30T16:43:40.000Z

@phoddie sorry total missed that, thanks a lot.

Answer 12 · 2019-01-30T22:05:06.000Z

@phoddie
Thanks for your help. Got it working. I had to modify your runmod a little. Adding some headers to allow JS to access the endpoint.

In main.js:89

-return {body: "done\n"}
+return {headers: ['Access-Control-Allow-Origin', '*', 'Access-Control-Allow-Methods', 'GET, PUT'],  body: "done\n"};

I made a little video showing it in action (click to open on vimeo):

To run it yourself go here.

Because netlify forces https you need to allow loading resources over http after hitting the Install/Uninstall button (little shield icon in the address bar).
All of this is currently only tested in Chrome 72

Going forward I think it would be best to write a tool that simplifies the xsc/xsl tooling in c so the emscripten overhead is reduced.

Answer 13 · 2019-01-30T22:15:49.000Z

@FWeinb - that is awesome indeed. Congratulations!

I will apply your HTTP header changes to the runmod repository for consistency.

Going forward I think it would be best to write a tool that simplifies the xsc/xsl tooling in c so the emscripten overhead is reduced

In watching your video and running the previous version of your page, performance doesn't seem to be a concern. Changes to xsc and xsl are going to be a significant undertaking to put in place, and then will need to be separately maintained as XS evolves. I would be hesitant to head down that road unless there are very significant potential benefits.

Would be you be wiling to share your code and build scripts? I'd like to be able to reproduce your build to be able to experiment further. Once we have some first-hand experience with XS on WAssm we will be in a better position to contribute to discussion of optimization.

Answer 14 · 2019-01-30T22:27:23.000Z

@phoddie
Thanks!

I will need to clean this up a lot. I have no idea of the codebase and just wrapped stuff in #ifndef/#ifdef until it compiled. I will work on that in the coming days. Currently it's a big mess.

But here is the basic idea I had:

Installed emscripten using brew (macOS only)
I added another architecture to xs/makefiles called js (maybe change that to wasm)
Modified the linux xsc.mk/xsl.mk to add emscripten specific stuff
Wrapped stuff in #ifndef/#ifdef. The emscripten build is very similar to the linux build. With the exception that signal isn't working so I just #ifndef it out.

One problem for building xsl is that it needs a platform dependent xsc to compile. That is currently hardcoded to use the mac build in the makefile.

I hope I can clean this up tomorrow so you can have a look at what I did.

Answer 15 · 2019-01-30T22:30:49.000Z

@FWeinb - Thanks, I look forward to checking it out.

Getting started with XS is a good challenge. ;) There's a lot there. You are off to a good start. We're happy to help find a way to make the wasm target maintainable moving forward.

Answer 16 · 2019-02-01T03:23:16.000Z

@FWeinb - thanks for the PRs. The changes do indeed look very small and straightforward. I'll have a chance to try them on the weekend.

Answer 17 · 2019-02-04T03:40:19.000Z

@phoddie Thanks!

I worked on this a little more and wasn't satisfied with having to compile xsc/xsl as standalone tools. The way emscripten is emitting code isn't optimal and I could not share a file system between these two build units. I had to copy files between xsc and xsl in the browser to link the various *.xsb files into one *.xsa.

Another problem is that the xsl wasm binary is ~1.2Mb big in an optimised production build. This is gzipped to ~400Kb but still a lot of code. Having two emscripten runtimes isn't optimal either.

During this weekend I build what I called wasm_xscl it is a very primitive merge of xsc/xsl into one tool that can be compiled with emscripten and is exporting the two functions compile and link. These are now sharing a filesystem and with this I was finally able to enhance the toy WebIDE by quite a bit. It is now possible to compile the ping example from the runmod repository.
You can test the WebIDE here

I know that having another tool will be a maintenance burden but I think that having a leaner more specialised tool for compiling will greatly improve the usefulness of the wasm port.
On the other hand I don't know what the directions of this project regarding this whole topic is, do you want to support this or would this feature just be "nice to have".

Answer 18 · 2019-02-04T07:36:51.000Z

@FWeinb - I spent some time on the emscripten build from your PR. It gets part way and then fails with what appears to be a Java version error. I tried with Java 9 an Java 8, but the error was the same. Any clues?

make
make -f xsc.mk
# xsc debug : emcc xsc
make -f xsl.mk
# xsl debug : emcc xsl
warning: undefined symbol: $SOCKFS
warning: invalid item (maybe a typo?) in EXPORTED_RUNTIME_METHODS: ALLOC_STATIC
make GOAL=release -f xsc.mk
# xsc release : emcc xsc
Exception in thread "main" java.lang.UnsupportedClassVersionError: com/google/javascript/jscomp/CommandLineRunner : Unsupported major.minor version 52.0
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:792)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)
shared:ERROR: closure compiler failed (rc: 1)
make[1]: *** [/Users/hoddie/Projects/moddable/build/bin/wasm/release/xsc] Error 1
make: *** [release] Error 2

Your latest WebIDE is a very nice step forward. It makes good sense to avoid having multiple copies of the wasm runtime, and to be able to share a file system between them. The question is how to do that in a way that we can support for the long term. That's not a small consideration, so it will take some time to sort out. We are, of course, interested in supporting a wasm build. You have unlocked a lot of potential with that. But first, we need to be able to build this here too so we can explore options too.

Answer 19 · 2019-02-04T07:54:07.000Z

@FWeinb - Figured out the Java problem. That gets further. Both xsc and xsl debug builds appear to succeed (there are xsc.js, xsc.wasm, xsl.js, and xsl.wasm files in the bin/wasm/debug). I will try running those tomorrow. However, the release build fails. Any ideas?

make -f xsc.mk
# xsc debug : emcc xsBigInt.c
# xsc debug : emcc xsCode.c
# xsc debug : emcc xsCommon.c
# xsc debug : emcc xsdtoa.c
# xsc debug : emcc xsLexical.c
# xsc debug : emcc xsre.c
# xsc debug : emcc xsScope.c
# xsc debug : emcc xsScript.c
# xsc debug : emcc xsSourceMap.c
# xsc debug : emcc xsSyntaxical.c
# xsc debug : emcc xsTree.c
# xsc debug : emcc xsc.c
# xsc debug : emcc xsc
make -f xsl.mk
# xsl debug : emcc xslOpt.xs.c
# xsl debug : emcc xsAll.c
# xsl debug : emcc xsAPI.c
# xsl debug : emcc xsArguments.c
# xsl debug : emcc xsArray.c
# xsl debug : emcc xsAtomics.c
# xsl debug : emcc xsBigInt.c
# xsl debug : emcc xsBoolean.c
# xsl debug : emcc xsCommon.c
# xsl debug : emcc xsDataView.c
# xsl debug : emcc xsDate.c
# xsl debug : emcc xsDebug.c
# xsl debug : emcc xsDefaults.c
# xsl debug : emcc xsError.c
# xsl debug : emcc xsFunction.c
# xsl debug : emcc xsGenerator.c
# xsl debug : emcc xsGlobal.c
# xsl debug : emcc xsJSON.c
# xsl debug : emcc xsMapSet.c
# xsl debug : emcc xsMarshall.c
# xsl debug : emcc xsMath.c
# xsl debug : emcc xsMemory.c
# xsl debug : emcc xsModule.c
# xsl debug : emcc xsNumber.c
# xsl debug : emcc xsObject.c
# xsl debug : emcc xsPlatforms.c
# xsl debug : emcc xsProfile.c
# xsl debug : emcc xsPromise.c
# xsl debug : emcc xsProperty.c
# xsl debug : emcc xsProxy.c
# xsl debug : emcc xsRegExp.c
# xsl debug : emcc xsRun.c
# xsl debug : emcc xsString.c
# xsl debug : emcc xsSymbol.c
# xsl debug : emcc xsType.c
# xsl debug : emcc xsdtoa.c
# xsl debug : emcc xsre.c
# xsl debug : emcc xslBase.c
# xsl debug : emcc xslOpt.c
# xsl debug : emcc xslSlot.c
# xsl debug : emcc xslStrip.c
# xsl debug : emcc xsl.c
# xsl debug : emcc xsl
warning: undefined symbol: $SOCKFS
warning: invalid item (maybe a typo?) in EXPORTED_RUNTIME_METHODS: ALLOC_STATIC
make GOAL=release -f xsc.mk
# xsc release : emcc xsBigInt.c
# xsc release : emcc xsCode.c
# xsc release : emcc xsCommon.c
# xsc release : emcc xsdtoa.c
# xsc release : emcc xsLexical.c
# xsc release : emcc xsre.c
# xsc release : emcc xsScope.c
# xsc release : emcc xsScript.c
# xsc release : emcc xsSourceMap.c
# xsc release : emcc xsSyntaxical.c
# xsc release : emcc xsTree.c
# xsc release : emcc xsc.c
# xsc release : emcc xsc
/tmp/emscripten_temp_BNewEQ/xsc.bc.o.js.mem.js.jso.js.jso.js.jso.js.jso.js:1: WARNING - Duplicate case in a switch statement.
(giant dump of minified JavaScript)
4 error(s), 2 warning(s)
shared:ERROR: closure compiler failed (rc: 4)
make[1]: *** [/Users/hoddie/Projects/moddable/build/bin/wasm/release/xsc] Error 1
make: *** [release] Error 2

Answer 20 · 2019-02-04T12:21:01.000Z

@phoddie Sorry for all these problems. I just updated to the latest version of emscripten and it looks like they updated the version of the closure compiler they use. I think it is fine to just remove the --closure 1 flag from the LINK_OPTIONS in both xsc and xsl.

Optimising these bundles will be done when building a web app using tools like webpack or parcel anyways.

Answer 21 · 2019-02-04T16:55:58.000Z

@FWeinb - This feels like getting started with a new embedded target: getting the build set-up is half the battle. Removing --closure 1 from the makefiles does indeed allow the build to complete cleanly. Thanks.

Answer 22 · 2019-02-04T18:25:52.000Z

@phoddie Just remembered. I had to add -s "BINARYEN_TRAP_MODE='clamp'" to the linking of xsl/xsc to get the build of the current ping runmod working. The problem seams to be that wasm is throwing an error and not clamping values.

Answer 23 · 2019-02-07T01:54:41.000Z

I updated the WebIDE to look more like a traditional IDE. You can load a gists and all modifications are persisted in the browser. This is far from perfect but we are getting there. Implementing Github OAuth should allow for saving/forking gists.

Finishing the log area and integrate the compiler warnings into the code editor to highlight the errors etc. is on my todolist

This is a very quick prototype, everything was build during ~~two~~ three evenings. Code ~~is quite messy~~ a little better.

Answer 24 · 2019-02-15T22:59:14.000Z

@FWeinb this is awesome, nice work! Any chance you'd be willing to share the code for the WebIDE? I'm interested in experimenting with it as a tool for some upcoming workshops.

Answer 25 · 2019-02-20T11:31:12.000Z

@lprader thanks for your interest.

I don’t feel good releasing this code before there is support to build the needed tooling implemented here. My recent PR was closed and I don’t know how long it will take to get a wasm build from this repo. I know this is will introduce a maintenance burden onto this project. Maybe @phoddie can help here?

Sorry but I don’t have the time to maintain a separate fork of this repository in addition to developing the WebIDE.