Dynamic import maps support
guybedford opened this issue · 45 comments
It seems like injecting import maps into the page after the first load is explicitly prohibited here:
Similarly, attempting to add a new <script type="importmap"> after any module graph fetching, or fetching of import: URLs, has started, is an error. The import map will be ignored, and the <script> element will fire an error.
This seems very restrictive to me. If the composition rule of import maps was to have them throw on conflicts, then the nondeterministic effects can be reduced here (at least for bare package imports).
This is something users will definitely request as soon as they start using this API. Sandboxes entirely rely on being able to do this sort of thing.
I'm not sure what you're asking for here. Once modules have begun fetching, we can't change mid-flight how module resolution works, so this constraint is very necessary. At the same time, it seems clearly documented. What is the action item?
Once modules have begun fetching, we can't change mid-flight how module resolution works
It could be possible to move from a state where import('asdf') throws for having no map, to a state where import('asdf') works out due to a map being dynamically injected. This can be a well-defined transition.
I know you have reservations on the determinism here but I would argue that this is an important feature to have, if not initially, that it will be needed eventually.
The action item would be to lay the groundwork for the above determinism by throwing when a package map name collides during composition.
The full behaviour for a deterministic dynamic injection would then be:
- It does not delay the resolver, only those package maps initial on page load delay the resolver
- Once loaded, it composes into the package map, throwing on conflicts with existing packages
- If there are any URL mappings, these throw as well as they are not supported for the dynamic case.
- On its
loadevent, animport()to any name in the new map can then be guaranteed to work.
That's a strange and crippled subset of the functionality that import maps provide; I don't think it makes sense to integrate it into this proposal. Rather, folks should investigate different workflows, e.g. server-generating their import maps. Import maps really aren't designed around dynamism, and it is a non-goal to support that; they have a hard enough job already solving all their existing goals without also taking on the role of a dynamic resolver system.
Import maps really aren't designed around dynamism, and it is a non-goal to support that; they have a hard enough job already solving all their existing goals.
This sounds more like an opinion than an argument. But yes, it does come down to discussing goals. I can tell from experience that this will be the top requested feature of import maps as soon as users use them.
No, it's more that if we had a different set of goals, we would have designed a different solution, e.g. one based on JavaScript functions instead of declarative JSON structures.
I suggest those interested in adding dynamic mapping to the platform work on doing that in a different manner, and not trying to extend this proposal to encompass it.
@domenic JS module loaders like RequireJS and SystemJS have provided dynamic mappings through a JSON mapping structure for years, so I don't think the design argument holds on that point here.
For whatever it's worth, I agree that dynamically adding an import map in the browser is a very worthwhile feature. Even after initial module loading has begun. I think it only makes sense for modules that are loaded after the initial page load (either via import() or via injecting a <script type="module"> after initial page load), but I find those use cases compelling.
I'm not sure if I understand all of the nuances of the argument about determinism, but it seems to me that if the user does import('my-lib') before they have defined my-lib in an import map, that that's quite reasonable as a user that my import doesn't work. And also that I'd like to import some modules with an initial import map before adding a new import map that has a few more modules.
Agreed with @joeldenning. If the browser has never evaluated import('some-lib'), then why would it be bad to interpret a new package map that defines some-lib even after other modules have been loaded?
To explain the reasoning here, in the draft import maps specification, HTML doesn't really keep track of whether it's done import("./some-lib.mjs") yet--what it keeps track of is what `"./some-lib.mjs" resolved to via the import map, and whether that was requested yet. I think it'd be a bit surprising if you had an API that observably changed what a particular import statement or dynamic import with the same argument does over time.
At the same time, it's true that the dynamism in this proposal is very limited. Although it's nice that you can use scripts to modify it, any use of <link rel="modulepreload"> will block further import maps from taking effect. It'd take a pretty opinionated page to be able to actually dynamically load more import maps.
I'm wondering what problems people have with this static design. My intuition is, if these things can be pre-computed, it could lead to faster page load time, so I like the design of the import maps proposal. What wouldn't fit in well to #92 (comment) ?
I think being able to load import-maps dynamically would be beneficial specially for SPAs.
We are currently using some sort of map for long term caching of static assets with RequireJS and the whole file would be huge if it was not split and loaded on demand.
Other concern I have is the performance penalty from fetching multiple maps at the start of the application if different parts of it were deployed in different servers or CDNs.
Finally some existing pages with complex setups heavily rely on adding some mappings from withing the application (i.e. environment/user dependent configuration) and being able to have a single module system is good for managing complexity.
Regarding:
I think it'd be a bit surprising if you had an API that observably changed what a particular import statement or dynamic import with the same argument does over time.
Maybe merging dynamically loaded maps could behave in a way in which the current one take precedence when coming across duplicated keys.
We are currently using some sort of map for long term cache of static assets with RequireJS and the whole file would be huge if it was not spitted and loaded on demand.
For me, that's the most compelling argument. In general, I can't imagine that you want to alter the way how imports are resolved. But you certainly want to extend it.
I'm a little unclear on the setup. How do you want to use import maps here?
I'm a little unclear on the setup. How do you want to use import maps here?
Imagine an app with some common modules to all routes and a big module foo which is loaded when navigating to /foo. Module foo has lots of sub-routes and requires lots of static assets so the import-map needed for it to work is quite big. I would like to be able to load foo import-map just before loading the foo module. If having a big SPA the amount of modules like foo could be quite big, thus resulting in quite a big import-map specially if we include information for using import: URLs.
Another use case would be for example for A/B testing or feature toggles: I may want to offer a given set of users a new version of some module while others keep using the old one, this is definitely doable without dynamic import-maps:
// foo.js
import new from './foo.new.js';
import old from './foo.old.js'
const foo = useNew ? new : old;
export default foo;But may be nicer using them.
In my current setup (nothing serious, just a PoC) I use import-maps for:
- Decoupling in general.
- Resolving the different app-modules though bare imports which allows for partial builds/updates only by updating the import-map.
- Implementing long term caching aliasing bare specifiers to hashed files. I.e. '@x/foo' : 'foo.asd123.js'
I don't have the need for dynamic import-maps here since I use rollup to bundle each module or sub-app and it takes care of dynamic imports and assets, but if I didn't use a bundler then a good number of entries would be written into the map which in most cases would be just fine but who knows how much could it grow.
I could definitely pass without the dynamic behavior of import-maps but I'm used to it while using different module loaders and I think it can be convenient.
@isidrok I'm missing something--why would the import map be dynamic, as opposed to what you dynamically choose to import with import()?
I'll try to be more specific:
- Reduce initial load time in cases where the import map has a significant weight.
- Allow to specify mappings at run-time conditionally without changing existing code. It's true that one could use dynamic imports and choose what to import but if that was not planned when the module was created then a static synchronous module would be turned into an asynchronous one requiring changes in its consumers.
All in all, I don't think its a needed feature but a convenient one.
Would it be accurate to sum up the issue like this?
If there are many import map entries potentially accessed by dynamic imports, this proposal is suboptimal because all entries are downloaded up-front, even those that are not used on initial load.
@littledan I would add:
- When different import-maps the application needs cannot be merged in advance (i.e. third party modules) initial load time could be penalized.
- Some module loaders allow dynamic configuration and people find it convenient sometimes.
Based on some related experiments, there's an alternative approach which may help achieve some of the goals in this thread (and solve other problems too e.g. for workers or Node) - without making it dynamic:
It might be better to spec a local map for each module in it’s metadata. The overall characteristics that arise are much better than a global map: for example, two modules could refer to different versions of a module, naturally avoiding dependency hell - without the additional scope mechanism, which tries to inverse the process and add module-local information on top of a global map.
The metadata could be populated by HTTP (link?) headers, co-locating the meaning of a specifier with the delivery of the module that uses it means you can have pay-for-what-you-use incremental loading, as opposed to having to download and parse what will eventually be an entire package-lock.json before you can run anything else.
In hindsight, this could still work, by taking the current import map as just one way to populate that canonical information that lives in the module metadata, enabling servers to also populate it via headers. I saw this was already discussed elsewhere (#1), but thought it was worth raising again as the trade-off's which pulled this strongly towards an application-level only approach may have changed and be better suited to extend in this manner.
See https://github.com/WICG/import-maps#scope; maps are intentionally global.
@domenic Do you think there might be a place for package-level maps? Maybe as source material for generating the global-level one? It feels like it would be useful for packages to bundle their own internal import maps for their dependencies (and maybe also their exports), and for that map to be part of what gets published to the npm registry.
Based on what we've seen so far in the ecosystem, I do not. E.g. you have a single app-level package-lock.json/yarn.lock. But, anything's possible in tooling land. I just haven't seen it.
Based on what we’ve seen so far in the ecosystem, I do not. E.g. you have a single app-level package-lock.json/yarn.lock. But, anything’s possible in tooling land. I just haven’t seen it.
Yes but each package has its own package.json. I think it would be useful as part of generating the global import map to know what the public paths are within each package. Like in your example for "/node_modules/socksjs-client/querystringify/index.js", what if that file is moved within that package or renamed? How would the global import map be updated to reflect that change, if not a user fixing it manually after investigating the new package structure? Whereas if socksjs-client could specify somehow what the path is to its querystringify export, then the export’s file could be renamed or moved without the package author worrying about breaking users’ import maps.
For Node we’ve been discussing this here, and maybe it’s something that Node just implements on its own, but it would be nice to coordinate the two since they’re obviously related if not interconnected.
In the end it's the application which decides what files are in node_modules, and on the web, it makes sense to me for the application to decide what paths are in the import map. I just don't see the value of the extra complexity you are proposing.
if not a user fixing it manually after investigating the new package structure
Precisely, a user (or more likely a tool the user is using, like npm or yarn) would become aware of the app's new package structure and generate a new import map. After all, most user's change their app's package structure via such tools.
See https://github.com/WICG/import-maps#scope; maps are intentionally global.
I’m not sure I fully grok, correct me if I’m wrong, but is your main concern essentially some XSS-esque scenario where a CDN changes the meaning of a specifier to be something malicious, and that wouldn’t happen with this application-level config?
In that case, I think you may be conflating goals and handicapping both. If you include a script in the page from somewhere else, that server can serve pretty much anything. The only way to lock that down would be SRI. Specifier-to-URL or URL-to-URL translation doesn’t change anything - even if it’s the application author doing it. If they map “jquery” to load from somewhere else, they could still get anything. This is probably more misleading to users that they are secure.
A better approach might be to separate out an application-level map for defining the hashes for the modules the page is willing to accept. That would be an actual package-lock for the web, and a way for app authors to lock things down. This isn’t a package-lock.
This way a CDN could actually legitimately fallback and serve that transitive jquery file from some other server. But the application doesn’t care, because it knows it’s getting the same file because of the hash.
@pemrouz no, the main concern is not XSS. The main concern is that application authors are the appropriate party to be in control here, not spread throughout their dependency graph.
Precisely, a user (or more likely a tool the user is using, like npm or yarn) would become aware of the app’s new package structure and generate a new import map. After all, most user’s change their app’s package structure via such tools.
I guess what I’m suggesting is that there be something for such tools to use to know how to generate the import map, for example to know what the paths within a package should be or point to. If there isn’t something defined in the spec for how this data should be stored within packages, each tool will come up with its own solution, and then packages might have a JSON file for npm and another for yarn and so on. I think we would be better off if it were standardized.
I don’t see how this spec, which is unaware of package.json files or even the concept of packages, would be able to address that. It might be a more reliable approach to get all the popular tools on board with the same joint standard, at which point (like broserify’s browser field) it becomes a defacto standard that everyone follows.
I welcome folks to work on inter-tool standardization efforts for how to generate import maps, and even use this repository as a way to coordinate while getting started. (See for example #108 or #60.) But this specification is indeed about a web platform feature, so the specification work here will be about import maps.
@pemrouz no, the main concern is not XSS. The main concern is that application authors are the appropriate party to be in control here, not spread throughout their dependency graph.
That was the other conflation I think, between "library" and "module".
For example, it would not make sense for a library to include a import map
My suggestion is not to have each "library" published with their own map.
It's that the meaning of the bare imports used by each module should be canonically stored in it's own metadata.
Even in the latter case, the resolution is done from the application-level perspective. It's not controlled by dependencies. It just means it can be delivered more progressively.
Those were the only two possible reasons I could understand from what you wrote. If you agree XSS is now not a concern, and app authors can still be in control, then you probably have additional valid context that leads you to a very different conclusion on the goals as @guybedford mentioned i.e. considering any progressive loading a non-goal, import maps a security mechanism, etc.
If other implementors share some of these concerns, then it would be more beneficial to discuss offline sometime, otherwise I don't have a strong opinion/desire to push this (/cc @littledan @annevk @MylesBorins). Small outline of what the counter proposal would consist of:
- Speccing a place in module metadata to store their own maps (this would also mean Node or workers can easily use the same)
- Keep the current format as a way to populate those maps
- Allow additional ways to populate those maps, such as via
<link>/Link - Spec a different out-of-band format for actually locking modules down using SRI. Agree and communicate that security is not a goal of specifier-to-URL or URL-to-URL rewriting and they should use that "module-lock.json" file.
I imagine we all share the goal that import maps not cause a lot of additional front-loaded fetching, and there are different thoughts on how big the map would be. Is this an accurate summary? How might we investigate further how big the maps would need to be in practice?
I'm not sure that I agree that there is consensus
I imagine we all share the goal that import maps not cause a lot of additional front-loaded fetching, and there are different thoughts on how big the map would be. Is this an accurate summary?
While I understand the need for the spec to be independent, those of us who are interested in micro ui frameworks such as single-spa would like a way to dynamically override the package map. Why would we want that? Because it's important that each of the teams that contribute to the larger spa have autonomy to use different versions of the same package. See this other thread: systemjs/systemjs#1860
For what it's worth, my game engine, Oozaru, would benefit greatly from being able to do this. One of Oozaru's goals is to be able to run the same code (unmodified) as the desktop engine neoSphere does; my setup includes a sandboxed file system where @/path points to a file in the game's directory--but this directory isn't known until the game is loaded. The engine has its own statically-resolved module tree of course, but that doesn't rely on the import map. The game's main module is loaded via dynamic import, which is when the import map is actually needed.
Right now I can work around the issue by hard-coding the game directory into the engine and writing the import map ahead of time but this isn't ideal: each game now requires its own copy of the engine, even though the game itself is loaded via dynamic import. Completely static import maps made sense when we only had static module resolution, but now that dynamic import is a thing, the lack of dynamic import maps is going to become a major pain point, I think.
SystemJS is going ahead with shipping this feature in its import maps support via <script type="systemjs-importmap"> as it is necessary for our users.
The PR was contributed in systemjs/systemjs#2215, and I have put together a specification outline of the approach in https://github.com/guybedford/import-maps-extensions#lazy-loading-of-import-maps and would be glad to participate in further specification work here.
@guybedford Interesting!
I'm considering doing something like this for three.js too.
Have you guys tried using serviceworkers for the mapping?
I was considering doing something like this:
<script type="threejs-importmap">
{
"imports": {
"three": "/path/to/three.module.js",
"OrbitControls": "/path/to/OrbitControls.js"
}
}
</script>
// Installs a service worker that remaps urls using the `threejs-importmap` in the page.
<script src="threejs-importmap.js"></script>
<script type="module">
import * as THREE from 'three';
import { OrbitControls } from 'OrbitControls';
</script>@mrdoob nice to see you are testing out these workflows. I'm not sure that approach can work since bare specifiers are not reflected to the service worker as far as I'm aware. Although perhaps a custom URI scheme might work? I'm not sure if so or whether that would be worth pursuing or not.
It really depends on how you position the workflow though. For example you can use import maps with SystemJS or ES Module Shims in a way that is not tied to a specific framework approach. SystemJS gives the best performance but ES Module Shims is pretty quick too.
The issue here is specifically about amending the import map after the first module load, which as of yesterday is a feature SystemJS supports through a custom dynamic import maps extension only.
I'm not sure that approach can work since bare specifiers are not reflected to the service worker as far as I'm aware.
Ah true... Before writing the post I tested doing import { Test } from './Test.js' to see if the Service Worker catched that and it did. But I just tried doing import * as THREE from 'three' and I got this:
😞
fwiw I worked on an import map service worker in https://github.com/joeldenning/import-map-service-worker. There are a couple issues with it - the first is that service workers aren't executed on the first page load before they have been installed. The second is that bare specifiers don't get forwarded to service workers. However, if you have interest in the approach feel free to look at that code for reference.
Just wanted to submit one more use case for dynamic import maps, the one we at Framer are facing. It's similar to what @fatcerberus described in application to his app, and I'm pretty sure every "online code playground"-type of app will eventually run into.
Framer allows users to write code that uses ESM for importing dependencies. We dynamically import() those code files and evaluate them in our sandboxed preview iframe. When the user updates their code, we import() the updated source again.
We would like to allow our users to define their own importmap.json that would be used we when import() their code, but given the current implementation of the standard we cannot allow editing that importmap.json without reloading the entire preview iframe, something we would like to avoid, because it's slow and suboptimal in terms of UX.
Dynamic import maps that, ideally, allow not only "appending" mappings but also "overriding" the original ones would truly save us, I recon.
We have micro service front end architecture.
We need this feature to load exactly the correct list of available modules. The list we get from server.
Is there an API to add entries to import map? like System.add('react', 'https://cdn.jsdelivr.net/npm/react/umd/react.production.min.js') ?
No, that API does not exist in the import maps spec, as far as I know.
I too was interested in exactly this functionality so I created https://github.com/keller-mark/dynamic-importmap (heavily based on es-module-shims -- thank you @guybedford for that great library!).
However I agree with all above who have said this needs to be standardized.
My use case is that I want to “externalize” big/shared dependencies like React while publishing an NPM package as ESM that is usable in both consumer packages (where node_modules resolution is available) and in plain HTML pages (specifically where i do not have control over all scripts on the page). Essentially I want the ESM equivalent of using UMD + script tags, but directly within the module and without using UMD.
My use case: A plugin system where the user is only able to write JS code, and that code is dynamically executed in a worker or sandboxed iframe. Some libraries (like @pixiv/three-vrm) require that you use an import map, which makes it impossible to just await import(...) it from e.g. jsdelivr if the plugin author isn't able to somehow specify the import map in their JS code.
Related:
Was pointed here after asking about this on mastodon.
Working on adding import map support to Drupal - a GPL PHP based CMS.
Drupal's rendering system stores the static parts of a page in cache and adds placeholders for dynamic parts (e.g. parts that vary per user).
Drupal sends the static part to the browser as soon as possible and then client-side code finds the placeholders and fetches those pieces and updates them in the browser.
When rendering Drupal allows developers to attach assets to markup - these are CSS and JS files. And we're looking to extend that to importmaps.
This works fine with the first render of the static parts, we collect any importmaps referenced by content in the static parts and send an initial importmap.
The use-case for dynamic imports is if any of the dynamic parts of the page also declare dependencies on importmaps we'd love to dynamically extend the existing import map to add new entries. We can do this in an orderly fashion, ie it can be the first task of updating the placeholders with new content, before any of the associated JS for that piece is rendered.
Is there an API to add entries to import map? like System.add('react', 'https://cdn.jsdelivr.net/npm/react/umd/react.production.min.js') ?
This is exactly the kind of API we'd love to be able to make use of. Although I guess it would be more like window.importMaps.add()
Obviously the interim solution for us is to just collect all importmaps (they're defined ahead of time so this is possible) and add them all to the page, regardless if any of the content needs them or not. But it would be nice in the future to be able to only add those we need to save on some bandwidth.
Thanks everyone for the discussions so far and for the current functionality - its awesome for a project like Drupal where many modules (Drupal modules, not npm) could rely on the same JS package and so we need to do bundling and code-splitting in an interesting way - by relying on externals and importmaps.
Came here to point out that browser-extensions cannot add module-mapped-libs to a page dynamically. Terrifying implications when I imagine the jank crypto-web-wallets must go through.
Digging around a bit and also see related issue here of people with issues that could be fixed by this. Weird importmap race conditions extend outside of just browser-extension use-cases.
At Shopify we've seen breakage when we moved certain scripts to type=module, due to the fact that some theme developers integrated import maps below those scripts.