Proposal: Load foreign code with ES Module Shims and import maps
thomashoneyman opened this issue · 5 comments
Try PureScript is going to have to be reworked when the PureScript compiler switches from CommonJS to ES modules for foreign code. This issue details a proposal for how to handle this rewrite.
Current Implementation
We currently parse through the generated JavaScript on Try PureScript to discover all dependencies present via require()
statements. For every dependency we find we do one of:
- Load the JS from
compile.purescript.org
(for example,Data.EuclidianRing/foreign.js
) - Look up its URL from the Shim.purs file which lists CDN locations for some common FFI libraries like
react
, and then load the contents
Once we know what dependencies to load, we take their contents and insert them into the code iframe. Finally, we load their contents via the evalSources
function in frame.js
.
Issues
This code is no longer going to work. PureScript 0.15 will only work with ES modules, so the code in Shim.purs
, Loader.purs
, and frame.js
will all need to be updated to work with ES modules instead of CommonJS modules -- in other words, there is no more require
or module.exports
; instead, there's import
and export
.
I also don't think we can instantiate the code the same way we do now, which I believe happens here:
trypurescript/client/public/js/frame.js
Line 25 in 53a82a6
Finally, if we swap all the way over to ES modules, we lose all Internet Explorer support, which we have historically not wanted to lose.
Proposal: SystemJS
We're clearly going to have to rewrite how we handle foreign code in Try PureScript. After a brief discussion with @natefaubion on the PureScript chat I would like to propose that we use two technologies:
- SystemJS, which covers IE11
- Import maps, which are supported in SystemJS, including for IE11
The SystemJS documentation on import maps does a good job of explaining how we can use this tool to replace our existing shim + loader + frame.js setup. If I'm reading correctly, we'd strip out almost everything and move the contents of Shim.purs
into an import map. We'd rely on SystemJS for everything else.
I've spent time trying to understand how client/public/js/frame.js
(I'll refer to this as the frame.js
) and the scripts in client/public/js/index.html
work (I'll refer to this as clientIndex.js
). Here's my current understanding:
- clientIndex.js creates the
iframe
element and tries to send to it the JS outputted by the compiler. It tries 100 times over the course of 10 seconds or stops as soon as frame.js notifies it that the code has been received. - frame.js loads its DOM, and then receives the JS from
clientIndex.js
. Once received, it notifies clientIndex.js that code has been received. Then frame.js executes themain
function from the outputted JS. - Whenever the user clicks on an
a
element in the iframe, frame.js will notify clientIndex.js of the click. If the click is to GitHub repo or gist, then clientIndex.js will allow the redirect. Everything else is blocked for security. See #137 and #140. - Each time the code is compiled/recompiled, a new
iframe
is produced.
So, I don't think we can drop frame.js
completely due to the GitHub link navigation handling.
As for SystemJS, I've looked at their react-hello-world example to understand how to use it.
Since this project uses the compile
end point to also show what the plain JS looks like, we shouldn't change what that outputs. But due to how SystemJS works, I think we need to expose an additional API endpoint (that I'll refer to as compileSystemJs
) that is the same as compile
but with a few more things. The workflow we need appears to be something like:
- on server setup
- compile package set's PureScript into JavaScript (
src
dir ->output
dir) - use Babel to convert the ES modules into SystemJS modules (
output
dir ->dist
dir)
- compile package set's PureScript into JavaScript (
- on
compileSystemJs
endpoint:- compile user's inputted code to an ES module
- use Babel to convert that into SystemJS
- use
esbulid
to bundle userland SystemJS code and package-set SystemJS code - send HTTP response of that output
- on browser side:
clientIndex.js
forwards the data received by thecompileSystemJs
endpoint to theiframe
iframe
sets up and manages whatever is needed for SystemJSSystemJS
loads the shims and executes the code
Per their docs, a module can be loaded either by a file or by a module name:
<!-- by file name -->
<script type="systemjs-module" src="/path/to/file.js"></script>
<!-- by module name -->
<script type="systemjs-module" src="import:name-of-module"></script>
We can't load the content by file since the server doesn't and shouldn't store any. But since the iframe
is recreated on every compile, we can reuse the same module name. So, I think we'd have to do something like this:
<!-- take the code received by clientIndex.js and
register it in SystemJS as a module -->
<script>
// not sure how to do this correctly yet...
</script>
<!-- then execute it -->
<script type="systemjs-module" src="import:main"></script>
The other thing to keep in mind is that the current code will call main
in the frame.js part. We would need to add a wrapper around the compiled output that calls main
before passing it to Babel, esbuild, and SystemJS.
First major issue I've encountered is that the plugin for transforming ES module syntax into SystemJS' register format doesn't work on the compiler's output.
Per the docs of SystemJS, we probably want https://github.com/guybedford/es-module-shims instead.
I got es-module-shims
to work on PureScript code. See https://github.com/JordanMartinez/use-ems Both decimal.js
and big-integer
work. I can't verify uuid
or the react
libraries because the corresponding PS code hasn't been updated to ES modules.
I don't currently have time to integrate this into Try PureScript and submit a PR that implements the needed changes. But I did want to give an update.
One other thing. We'll need to use a CDN that provides NPM deps in the ES module format. We're currently using unpkg
, but most modules there seem to be UMD, which don't work with es-module-shims
. Per es-module-shims
docs and example, we could switch to https://jspm.org/, which can be easily used to generate an import map for the shims. See the repo's dist/index.html
for the map and other HTML it generated painlessly.
CC @mikesol
Thanks for the update!
I'll see if I have time next week to make a PR with these changes.