Feature: Named exports when importing CJS
giltayar opened this issue · 108 comments
Currently in NodeJS, if I import from a CJS module,
import ns from `./cjs-module.js`It will allow this, but won’t allow named exports, i.e. the following is not allowed:
import {namedSomething} from `./cjs-module.js`.This is called transparent interop.
The reason that NodeJS doesn’t allow named exports is that determining what those exports are MUST happen in the parsing phase of the ESM Spec (according to some, although there is contention about that too), and executing code is not allowed at or prior to that phase. But a CJS module’s exports can only be determined by evaluating the code in it (static analysis of JS is not an option as it is not determined to always give the correct results).
This would maybe have been cool if NodeJS was the first ESM implementation, but people are used to the babel way of doing modules, and in the babel world, named exports from CJS modules are allowed. This is because babel does not conform to the ESM spec to the letter (it can’t, because it just transpiles ESM to CJS).
And, good or bad, developers expect to use named exports when importing CJS.
I see these options:
- Continue the existing way (no named exports for CJS)
- Don’t conform to the spec when importing CJS
- Do late-linking/shaping of named modules based on late evaluation of CJS module
- Disallow transparent interop, and enable
import.meta.require(or equivalent) to enable importing CJS from ES modules - Enable metadata in the CJS module that can statically describe the exports for ESM, e.g. something like
//export default; export foo, bar;at the head of the CJS file, thus enabling named exports when importing the file.
I am sure there are others options, so if you have another option besides those four, please add it here.
It would be great if for each options you specify pros and cons, or at least if you don’t like the option, specify a clear and simple use case that would be problematic if the option was chosen.
Edit by @GeoffreyBooth: Use case 12.
It’s worth reading the comments in #80 too, especially concerning the observability (or not) of what gets done in the parsing phase.
I would encourage us to try to find a solution that threads the needle if at all possible. This probably isn’t a stark decision between honoring the spec or not. I bet we can find a solution that enables this while still adhering to spec, or adhering to the spec as far as any user-executable code could ever know, or adheres mostly to spec. I think it’s worth exploring the gray area as much as possible.
personally I'm not a fan of importing cjs at all. (as a bonus we could drop the mjs extension if we didn't support it.) I would prefer something like import { makeRequire } from 'module'; const require = makeRequire(import.meta.url);
A very large number of the use cases we've documented require being able to import 'cjs' and require('esm') - that's probably one of the more contentious issues overall, of which "should import 'cjs' support named imports" is a small subset.
I think spec compliance is paramount, and I'd love to have import { Children } from 'react', for example, work (where "react" is a CJS module) - but I suspect we'd have to come up with a way to get consensus to change the spec for that to become tenable.
Some of the discussion in #80 revolved around how to define “spec compliance.” If external code can’t see what Node is doing under the hood, and if Node is compliant as far as the external code can detect, than can that be considered “spec compliance”? Because if so, then that’s a way to have it both ways: do whatever parsing or evaluating that needs to be done in the parsing phase, in a non-observable way that has no side effects, and you get the ability to import CommonJS named exports without violating the spec. The trick is figuring out that “non-observable way that has no side effects.” But I think that’s a straightforward engineering problem that can be solved.
I think we should bring this question to TC39.
@GeoffreyBooth i believe that it's always possible to write code that observes the ordering of evaluation and linking, which is why it would require a spec change in EcmaScript to have node do that and be compliant.
@mcollina I believe @bmeck has already done so, but if there's new information it might be worth another shot.
I'm cool having CJS evaluate before resolving named exports. It's what we're doing for builtins modules today (with the assumption errors/side-effects won't happen during evaluation). For the broader case, of more than just builtin modules, if errors do happen during evaluation I don't find the difference that ghastly, since that is something CJS users expect. The concern is scoped to CJS interop, so isn't something that bleeds into browser interop scenario, and can be something that is opted-in-or-out of as needed without broader approval from others like the TC39.
That’s another good point—could we do this evaluation during the parsing phase for CommonJS imports only? Because the spec doesn’t concern itself with CommonJS, right? So however Node wants to handle CommonJS is up to us, so long as we follow spec with regard to true ES modules?
i actually proposed this way back in my first pr (nodejs/node#16675) and TC39 discussed it here: https://github.com/rwaldron/tc39-notes/blob/master/es8/2017-11/nov-28.md#9iie-discuss-module-order-instantiationevaluation-guarantees
its worth noting that (afaik, please correct me if i'm wrong) source text refers to whatever the original thing is that rules what is exposed from the import, which is in this case the cjs
@GeoffreyBooth there have been a variety of approaches implemented and or looked at but they seem to fall into 3 real areas of investigation.
-
nodejs/node#16675 which did out of order evaluation.
-
https://github.com/bmeck/node/tree/named-export-core did pre-parsing to form the shape instead of evaluation.
-
https://github.com/rwaldron/tc39-notes/blob/master/es7/2016-09/sept-28.md#es-modules-lifecycle discussed late linking (shape of dependencies not verified until evaluation reached).
As far as I know the user-land esm loader works with named exports of CJS modules in ESM without order of evaluation issues or pre-parsing CJS export pragmas (named-export-core). This means that the root issue may be something deeper like being able to perform creation, instantiation, and evaluation phases synchronously.
std/esm i think falls into the third category of having an unset shape until evaluation time.
Ah, the third category could be closest. It seems like our current --experimental-modules named exports support is close to that as well since it creates the ESM projections (named exports, namespace objects) based on the fully evaluated CJS module. (Synchronous phases may make it mesh better though). This is something doable, with the lower level bits available, today.
It will likely come down to if enough folks think the order nit for CJS interop is worth not having it, opting-in-or-out of it, or worth creating more complicated workarounds for it.
It's not just scoped to CJS interop; it's something that applies to anything that can be imported, including JSON - although wasm and HTML imports are likely to have a form of static named imports, the ES spec still requires that the names be statically available. It's not just about errors - the order of side effects is impacted as well.
If we can get CJS interop for named imports (transparent or otherwise), that would be excellent, but I don't think spec compliance on evaluation ordering is something node core can or should lightly toss aside.
It's not just scoped to CJS interop;
Interop, and any affordances made, can be scoped as needed.
WASM and HTML imports don't have an existing rich Node interop history (as Babel+CJS+ESM does).
If we can get CJS interop for named imports (transparent or otherwise), that would be excellent, but I don't think spec compliance on evaluation ordering is something node core can or should lightly toss aside.
CJS interop, however it's handled, will likely need to make affordances for some things (like the order concern). Being open to them in some form or another is a good thing.
evaluation ordering prevents polyfills and other things from producing side effects before out of order evaluation that are hoisted before them. it is not a light allowance or affordance to be treated as good by premise alone, please see the PR above by @devsnek
Completely wonky module order as with #16675 isn't desirable but it's not all-#16675 or nothing. As mentioned, there are ways to approach things so that level of mismatch is avoided. It's a balance between expected CJS and ESM behavior for sure. Digging into working implementations, like that of the user-land esm loader, to suss out requirements, limitations, and considerations may be a good first step to see what all would be needed to enable named exports to more than just builtin CJS modules.
@jdalton it isn't as if your loader is doing something completely undiscussed, we can look at it but need to discuss the topics at hand like we laid out above in PRs and links rather than example implementations that require spec changes for that late binding behavior if we are unable to even agree on if spec violation is desirable.
re: @MylesBorins AFAIK, There's nothing concrete to ask for because as long as you don't consider cjs modules as having esm semantics (they do not), the only place where the spec (as written) is potentially broken is including cjs in the graph at all (because it might create circularities, which are expressly forbidden - see the spec edit needed for wasm modules). I remember reading some notes from one of talks where they talked about execution order - esm execution order is supposedly well-defined by the design of the dependency graph traversal. However for non-es modules which do not participate within the graph in the same way, well, they aren't actually be specified, since they're not es modules (they just have to provide a list of exported names, by whatever means necessary). The final spec requirements for the runtime semantics of resolution, in the end, are simply:
- The normal return value must be an instance of a concrete subclass of Module Record.
- If a Module Record corresponding to the pair referencingModule, specifier does not exist or cannot be created, an exception must be thrown.
- This operation must be idempotent if it completes normally. Each time it is called with a specific referencingModule, specifier pair as arguments it must return the same Module Record instance.
There's nothing that actually states how any non-esm source-text should or should not be executed. The idea that no execution of any js at all should happen during resolution is seems like a fabrication that entered the group's zeitgeist at some point which I cannot find a concrete source for (most all restrictions are placed on SourceTextModuleRecords, which a cjs module should not be, as it is not esm). Someone who believes that is the case may be able to cite the relevant spec points (at which point they can be campaigned to be changed)? But AFAIK there are none. How's the spec to enforce that a host doesn't execute anything for a native module upfront? As far as I can tell, it really doesn't.
@weswigham Abstract Module Records have specified semantics in the spec but we also discussed exactly what you are trying to avoid in TC39 as @devsnek also mentioned in relation to his PR https://github.com/rwaldron/tc39-notes/blob/master/es8/2017-11/nov-28.md#9iie-discuss-module-order-instantiationevaluation-guarantees . There is agreement that this guarantee is intended but not easily phrased in concrete language, we can create the concrete language if someone seeks to violate this without going to TC39 was my understanding from that meeting. The effort of creating such language is non-trivial and the idea that someone is going to treat Abstract Module Records out of order was expected to be somewhat out of expectations.
There have been other discussions in the spec and related bodies around ordering and why it should act a specific way as I aggregated for a slide deck in November 2017 https://docs.google.com/presentation/d/1RXvvScD8ce2FyLY2aYhbas83WCiBqzIOqdMt4OpkCJM/edit#slide=id.g2a9c910b7b_0_10
Per your points above:
However for non-es modules which do not participate within the graph in the same way, well, they aren't actually be specified, since they're not es modules (they just have to provide a list of exported names, by whatever means necessary).
They are leaf nodes that still call evaluate at a specific time as Abstract Module Records.
Someone who believes that is the case may be able to cite the relevant spec points (at which point they can be campaigned to be changed)?
See the presentation about this topic from November / how https://tc39.github.io/ecma262/#sec-innermoduleevaluation calls module.Evaluate() on non-SourceTextModule records at a specified time.
There's nothing that actually states how any non-esm source-text should or should not be executed.
That is outside of the scope of the JS spec for CJS, but the intention is to have a specific behavior as I described in my preamble above.
How's the spec to enforce that a host doesn't execute anything for a native module upfront? As far as I can tell, it really doesn't.
See the conclusion about it being hard to concretely phrase even if that is the intent.
Beyond this claim that there is no explicit ban on the behavior, we could add the ban at TC39 if required or we could change the timing of when module.Evaluate() happens.
it isn't as if your loader is doing something completely undiscussed,
Bits and bobs have been discussed in isolation though the lens of the limitations/confines of the existing Node WIP implementation and PRs. However, I don't think the combination of approaches that create a working implementation has been dug into. Since there are working implementations, they can be used to suss out requirements (whatever they may be). Once we identify what it would take we can bucket what we think would or wouldn't need specification changes.
@jdalton we have pointed out a specification change above that is needed to match your loader's semantics. We haven't even talked about other loaders that I can tell.
@bmeck while I appreciate the at-a-glance takes I think a deeper look would be a good thing. The jump to required spec changes also seems like it's skipping a few steps and assumes everyone is on your same page regarding what is and what isn't acceptable for CJS interop without requiring a change to the entire language specification.
@jdalton I disagree as these have been discussed, I have read esm several times, and I am pointing exactly to what esm is doing and related existing discussions for people to read. I have noted that your statements tend to avoid discussing the links I provide and the points contained within; as well as avoiding stating that the discussions are either irrelevant, incorrect, or relate to your implementation. Hence me consistently bringing up the topic of interest related to esm and relevant materials.
@jdalton I would say that perhaps discussing named exports itself is jumping the gun. We haven't reached consensus that importing common.js is something we want to support as "an official default loader".
I think everyone is on board with having named exports for a common.js loader, the requirements change depending on if it is official and default
May I suggest that we hold off getting too deep into the implementation details before we have the requirements?
Abstract Module Records have specified semantics
Yep. Not execution semantics, though - just a required interface.
There is agreement that this guarantee is intended
Within esm. It is impossible for the spec to specify non-esm modules' execution semantics. esm modules even have the potential to load in about the reverse order that someone expecting a cjs-style load could get (namely, your deps execute completely before you do, whereas in cjs you're free to execute before you require and thereby execute your dep).
The effort of creating such language is non-trivial and the idea that someone is going to treat Abstract Module Records out of order was expected to be somewhat out of expectations
Abstract module record only exists to support host-defined module formats. Like wasm. Or native. Or cjs. The execution semantics of these (not usually even js) modules cannot be specified (without overstepping into a host's domain). That's why it's not specified.
call evaluate at a specific time as Abstract Module Records.
And what evaluate does for non-esm is arbitrary. For cjs, it can just return a cached object or throw a cached error and be to spec. Nothing about the name evaluate actually enforces that any evaluation must occur when it is called for non-esm module records. It may as well just be called "get namespace", which would also be more accurate, since per spec it'll only even evaluate an esm module once anyway - every other call must return a cached namespace object.
See the conclusion about it being hard to concretely phrase even if that is the intent.
You could insert text along the lines of "No source should be executed within a realm during the resolution or instantiation phases of module resolution (eg, the realm is temporarily frozen, all evaluation throws)"; but it definitely doesn't say that today, and I would argue heartily against it as it would prevent a native module from using anything in the runtime. It's not there because it can't be reasonably said.
What I think core core takeaway from that discussion, just from the notes, wasn't that cjs shouldn't be executed whenever (you can't actually prevent that), but rather that it shouldn't interfere with how esm should execute (which is an ok guideline).
I disagree as these have been discussed, I have read
esmseveral times
Parts have been discussed. However, I think a look at it as a whole could be a good thing. If you feel you've exhausted your review that's cool. No one is saying it's all on you. A working implementation without engine/standards tie-ins that accomplishes CJS interop that "just works" is something that should totally be studied is all.
I would say that perhaps discussing named exports itself is jumping the gun. We haven't reached consensus that importing common.js is something we want to support as "an official default loader".
FWIW I don't think this (named exports of CJS) has to be the default either. I'm totally interested in named exports of CJS. But I'm interested in it as any option. If that's through something like #82, that's cool too.
The execution semantics of these (not usually even js) modules cannot be specified (without overstepping into a host's domain). That's why it's not specified.
@weswigham all the browsers (the hosts) in the room when it was discussed agreed that it was absolutely intended to be specified, it's just that nobody knows how to word it properly. node can certainly choose to deviate here, but regardless of what rule lawyering is applied to the spec text, the evaluation ordering semantics of all modules, esm or not, is indeed either regulated or intended to be regulated - and if one engine finds a loophole, the spec will simply be immediately modified to close that loophole. Trying to be pedantic about the actual spec text when the explicit (still perhaps unwritten) intention of it is clear, isn't going to help us achieve anything.
@ljharb The TC39 mentions by you and Bradley (quoted below) may seem kind of harsh, almost obstructionist, to those not familiar with the spec process. Could you provide a bit of context to explain why things like
we could add the ban at TC39 if required
and
and if one engine finds a loophole, the spec will simply be immediately modified to close that loophole
are part of the process.
From my perspective as a user, I (and tens of thousands of others) have been writing code like
import { something } from 'commonjs-module';since 2015. That’s a lot of code. We’ve been good, and we followed the correct standardized syntax we were told to use. The reason we used that ES2015 syntax, even though it required transpilation, was that it was supposed to be the standard that we could rely on, that was future-proof. That’s the “spec” that most JavaScript developers are familiar with. That’s what I as a user expect Node to support.
Sure, I agree that “spec compliance” isn’t something that we “should lightly toss aside.” But from the perspective of most users, if Node doesn’t support an import statement as we’ve come to understand it, it’s Node that is out of compliance. Maybe users are wrong to think that, from your point of view, but they’re our users and ultimately it’s our job to give them what they want, as reasonably as we can.
@jdalton oh sure!
The consensus in the room was that violating execution order consistency was a terrible thing, and the committee was largely shocked anyone would consider it. The consensus ended up being "we definitely want to ban that, but we're not sure how to word it", and since node at the time indicated it would not be violating that ordering, there hasn't been a rush to come up with wording.
@GeoffreyBooth yes, and it's very very unfortunate that ESM syntax was widely adopted long before node had figured out interop and before the browsers shipped implementations, but that's a hazard of the old TC39 staging process that thankfully won't be repeated. There was never any guarantee of the syntax being "future-proof"; and many pointed that out - but those who did, myself included, were ignored because the new syntax was shiny.
This leads me to think that Node CJS interop may not need to be a core TC39 specification concern. If CJS interop is to be a thing the interop for CJS may technically be breaking (in one area or another). The extent of the breakage and the ramifications of it should be looked into though.
@ljharb could you create a series of examples (maybe in a user repo) that demonstrate potential problem scenarios (load order, side-effects, etc.) that we can reference when considering approaches?
we all are all really focused on this one method of providing cjs named exports because (not sure really, easiest way maybe?) but it's pretty clear that some people think it's not safe. I find that concerns about unsafe should usually trump people saying it's safe.
I would encourage everyone here to think of other ways we can achieve our goal that doesn't use out-of-order evaluation, as this current argument has gone far past pointless
I added "late linking" as an option in the issue description. I believe it's a valid option.
Could late-linking be the silver bullet of named exports for CJS? While the spec says that we need to finalize the shape of the module (as described in the Module Record) int the parsing phase, what if we finalize it for CJS modules only in the evaluation phase?
And if we late-link, the order of evaluation is still conserved!
I'm guessing it could be done, but we would be losing something in the process: live bindings when importing CJS. But that would be, IMHO, acceptable, given that current CJS does not provide for live bindings.
If this is true—if we can late-link a CJS module during the evaluation phase, at the price of losing live bindings for CJS—it may be an answer to all our incompatibilities with the current babel-shaped ecosystem, and a good answer to other dynamically-shaped languages (like wasm?).
I'm guessing it could be done, but we would be losing something in the process: live bindings when importing CJS.
Be glass half full. The user-land esm loader avoids module load order issues, has CJS named export support with live bindings, and is interop configurable... which means all of it is possible. The issue raised by some here is trying to fit the what's-possible into a box of also-totally-spec-compliant, which isn't necessarily a box it has to fit in for all scenarios/configs.
On the better-fitting-in-the-spec-box front we could look at what's possible, with working implementations, for this current feature and suss out the parts of the spec we think need updating to achieve a better fit. If some feel user-land solutions carry less weight a more core centric implementation variation could be attempted (as was done with experimental named exports for builtin modules).
@jdalton we are well aware of your endorsement of your loader.
(as was done with experimental named exports for builtin modules).
I think there is a misunderstanding here. The semantics and visible API for builtin modules is spec compliant and was created that way intentionally and only after being created with compliance was non-visible optimization introduced. The semantics and behaviors are compliant which is different from late binding which is not spec compliant behavior and cannot be recreated with compliant ESM. As I have stated before, non-visible optimizations by compilers and vms are often not spec compliant in order to achieve better optimization, but always ensure the optimizations comply to semantics of the specification. This is not the same as blatant creation of semantics in conflict with specification.
We should not compare optimizations that match spec semantics and have non-visible spec breakage with ones that have visible violations of the spec.
Working reference implementations are super useful for proofing things out and giving heads-up to possible issues.
I think there is a misunderstanding here.
No misunderstanding. The example provided was of a user-land solution that transitioned into a core implementation, which some may feel carries more weight. Regardless of perceived persuasive weight another attempt is a good thing. Using the previous example (experimental named exports of builtins), it's development in core shook out issues and edge cases not considered before.
@jdalton we are well aware of your endorsement of your loader.
I'm confused, jdalton wasn't endorsing his loader anywhere as far as I can tell nor was he ever suggesting it as a way forward. I read it as just communicating that the user expectation is that this will "just work" based on the use cases we discussed related to this issue.
I think that it's very important to meet that user expectation or otherwise to solve the pain users will undoubtedly have with it.
Even if we don't allow
import { first, other} from 'someCjsModule
I think the consensus in #70 is that "what users expect will be harder than what they currently have in userland solutions" isn't great.
Given that we:
- Don't want to deviate or break the ECMAScript spec at all and want to ensure we are spec compliant.
- Don't want to ship an implementation that will surprise users when they use CJS.
Do you see anything that can be addressed either from the Node or spec side? Some examples could be:
- Some metadata that can be added and perhaps automatically generated to make named exports for CJS modules.
- Some convention or tools we can use to get named exports, maybe a command line Node.js flag that adds this behavior with a warning.
One thing that is clear to me is that we've failed (not sure what 'we' I'm talking about here) in educating the community about how named exports work and the difference between them and destructuring.
If you'd like data to back my intuition of users expecting import { foo } from 'someCjs' to "just work" I will gladly do my best to collect some.
Don't want to deviate or break the ECMAScript spec at all and want to ensure we are spec compliant.
Some, like me, don't mind deviating under certain configs/scenarios. Preserving the spec-as-is-today and CJS interop are at odds which leads to less-than-great-contorting to get things to kind-of-mesh. Something's gotta give. Either spec amends/updates or concessions for bits that don't align.
TBH, if we cannot find a way to offer an expected experience with CJS in our default loader that is both what developers expect and compliant to the spec seems like a good reason to not offer cjs support in our default loader. I have a number of other reasons to push back against transparent interoperability, but have avoiding digging into them due to our current yes and approach.
I'll once again encourage folks to not dig too deeply into this at the moment. If we do not have consensus on transparent interop, we don't yet need to dig so far into this. As mentioned above, CJS + transparent interop could be offered by a non default loader which would not ned to have the same level of compliance.
@benjamingr I see it as solicitation at this point; there is a distinct lack of discussion of topics about how it works and claims that it "just works". There needs to be more discussion about what it is doing which I have tried to bring up several times, but repeatedly just see replies about how it "just works" without context to any of the routes that have been discussed related to it and how it relates to the specification. Even in other meetings about userland as a whole there has been repeated lack of discussion of other loaders and it seems intentional at this point to exclude topics relating to potential discussion about problems/concerns and exclusion of other loaders. If there was more discussion about how/why the loader works rather than statements that it "just works" I would feel it to have viability in this discussion. As it stands, there is little in the way of talking about the user/mechanics of esm and more generic statements about esm itself without direct relation to the topics of concern in this issue. It seems like a sales pitch.
I think the consensus in #70 is that "what users expect will be harder than what they currently have in userland solutions" isn't great.
Agreed, but as I've stated we need to discuss what changes need to be made to accommodate things, not just tout some loader that has been pointed to have semantics that cannot be created to spec. We can talk about changing the spec, but talking about how something "just works" doesn't relate it to if it is allowed by ESM as it is specified today.
Do you see anything that can be addressed either from the Node or spec side? Some examples could be
Yes, late binding as discussed in the links above is possible to add the specification. It adds a codegen complexity and removes some static analysis from ESM though. You do keep live bindings just like how @devsnek 's PR for builtins shows, albeit at some synchronization caveats.
Some metadata that can be added and perhaps automatically generated to make named exports for CJS modules.
This was looked into by myself and some on Babel, but @jdalton was quick to harp on speed issues in our initial implementation as he was posting praise for esm at the time, and I stopped research in part due to the pressure he applies to people when he pushes his loader as superior even in the alpha phase of research into alternatives.
@jdalton people like myself are not willing to accept spec violations without consent from the TC39 body that the violations are future safe and/or spec changes to allow the behavior. See WASM's efforts to change the cycle problem for non-SourceTextModuleRecords for an example of changing the specification instead of violating it.
@benjamingr and @jdalton—while I opened this issue, and am well aware of the HUGE mass of existing code that assumes that transparent interop with named exports works—I want to point out what I believe this transparent interop leads to: the .mjs (or any other out of band mechanism) as a mandatory way to determine whether the file is a module or a script.
Why? Because if we have transparent interop, then the import statement can import both CJS and ESM, and we need a mechanism (which does not include parsing of the file...) of determining which is why. Thus, .mjs as such a mechanism (but other mechanisms are OK, such as @guybedford's mode=esm flag).
OTOH, if the caller determines what kind of module it expects (CJS or ESM), i.e. import for ESM and import.meta.require for CJS, then we can safely keep the .js extension for both (although, as @devsnek pointed out, we may want to keep the js/mjs split for tooling purposes).
I am trying to keep a neutral voice and not specifying what needs to be done, but rather analyzing what the implications of each decision we make are.
Can we maybe not allow transparent interop, and require everybody to use import.meta.require to import CJS? Maybe. Having babel/webpack/ts support it seems trivial, but there is a non-trivial amount of code out there to support.
If we do not have consensus on transparent interop, we don't yet need to dig so far into this
But to determine whether we want transparent interop or not, we need to understand whether we can allow named exports in transparent interop. If there is no valid way to do that, then transparent interop becomes MUCH less useful, and will maybe be dropped only for that reason.
And, yes, I know that there are other reasons to drop it, as I said talked about my above comment. But any input we can get on that matter is important.
I'll once again encourage folks to not dig too deeply into this at the moment. If we do not have consensus on transparent interop, we don't yet need to dig so far into this. As mentioned above, CJS + transparent interop could be offered by a non default loader which would not ned to have the same level of compliance.
Rock!
Update:
If we do not have consensus on transparent interop,
If consensus on transparent interop is a prerequisite for this feature we should capture it as a first-steps and we should make consensus on transparent interop an agenda or roadmapped item.
@giltayar the caller can’t always know what kind of module it is, because there can exist modules that could be both CJS and ESM.
Without transparent interop, the ecosystem will fork into as many shards as there are popular custom loaders. We should definitely try to achieve consensus on this ASAP. (Hopefully the meeting after TC39, instead of the one during it)
Regarding named imports from CJS, the current spec situation seems to be that:
- CJS modules are not "Source Text Module Records", since they are not ES modules
- Since they are not "Source Text Module Records", their loading and execution behavior are largely unspecified, but
- We want to execute CJS modules at "Evaluation" time in order to have consistent module execution order
Also:
- We don't know the "names" exported from the CJS module until we execute the script
- When a CJS module is requested via
HostResolveImportedModule, there is no information provided to the host about which names are being requested
Let's suppose for a minute that HostResolveImportedModule was supplied with an additional argument: a list of names being imported from the requesting module.
- A "CJS Module Record" could then be returned from
HostResolveImportedModulecontaining all of the requested names - All names would remain uninitialized (TDZ) until module evaluation
- During evaluation, the CJS module script is executed and then
module.exportsis used to initialize each exported name defined in the "CJS Module Record"
Thoughts?
@zenparsing that seems like a liberal interpretation of the spec that doesn’t match its intention; there would still be concerns there about side effects ordering, and i still think the overall committee would prefer to change the spec to disallow that interpretation over waiving those ordering guarantees.
@zenparsing Unfortunately the first time a imported module is encountered, we might not yet know all requested imports. E.g. we might pass { x, y } as the only requested imports but then run into another module deeper in the module graph that also tries to import { z } from that same module.
@ljharb I agree that we definitely want to preserve consistent execution ordering between CJS and ESM modules. In the thought experiment above, no CJS code is executed prior to the evaluation phase, so I think we're good on that point.
@jkrems Good point. To make it more concrete:
a.js
import { x } from './c.js';b.js
import { y } from './c.js';Extending the thought experiment, HostResolveImportedModule would have to return distinct Module Record "images" for each import of "c.js" found in "a.js" and "b.js". Let's call them CJSA and CJSB.
- Calling
GetExportedNamesonCJSAyields["x"] - Calling
GetExportedNamesonCJSByields["y"] - Calling
Evaluateon either one does something like:- If the underlying CJS script has not been executed, then execute it and cache the
module.exportsvalue as well all properties found on that object in a dictionary. - Populate the module image lexical environment using the cached dictionary.
- If the underlying CJS script has not been executed, then execute it and cache the
I don’t understand all the mentions of going back to the TC39 committee to “add a ban” or “close a loophole.” It feels to me like this is happening:
- Users submit a feature request.
- Crafty engineers find a way to satisfy the request without violating the spec.
- Others rush back to the committee to amend the spec to disqualify the new solution.
This feels broken to me. We’re supposed to be working to satisfy users. If the spec doesn’t allow a particular solution, or if people have concerns about a solution with regard to the spec, shouldn’t the trip back to the committee be for the purpose of finding a solution that works in harmony with the spec? To do otherwise feels like working against the goal of satisfying the feature request.
@GeoffreyBooth “users” includes many people beyond node, and there are critical use cases being defended that some “feature requests” would block. In other words, “some people want something different” is not sufficient to override all other concerns.
Absolutely if a solution can be found, we should pursue it! Keep in mind tho, not all feature requests should be granted, on any project - tradeoffs and constraints have to be weighed, including (for TC39) those of non-node users as well as node users.
@GeoffreyBooth this is part of why the term "rule lawyering" was mentioned earlier. The lack of spec violation is being done by trying to find loopholes or underspecified behavior in regards to order of evaluation. Identification of this underspecified behavior was brought to TC39 to see what the intent of the committee is on the issue and if such is considered underspecified / would be desirable to prohibit. Per the conclusion at TC39 in November 2017, the change to explicitly prohibit such behavior is intensive and was not thought to be necessary because at the time such loopholes were not being sought for actual use. Creativity is good, but we need to be careful about when creativity is encouraging something that might not be sound or future proof.
To my knowledge we have not had any direct discussion on late binding at TC39 except mention of it as a possible route to take in September of 2016, but that would require specification changes in order to be able to be compliant, and some engine work for VMs. I have had personal discussions about it on IRC previously but nothing except rudimentary questions about implications for VMs and discussion of static analysis regarding it.
@bmeck I was not part of any previous interaction and I'm definitely the least knowledgeable about the inconsistency of loading modules with exports that need initializing first (other than reading some interesting discussion in the node PR for builtins and the 'top level await' stuff).
I feel like there is a lot of previous baggage here and I'd like to see how we can help move forward in a constructive way. I realize that given the tremendous amount of work you've put into this and the context I (and others reading here) are missing it might be extremely frustrating on your part and I'd like to know if there is anything I can do to help with that as well as thank you for it.
What do you see as a path forward that would enable:
- Spec compliance as well as communicating concerns to TC39 and collaborating with the committee in good faith.
- Giving users the ability to solve the discussed use case brought in the use case meeting - namely to be able to use named imports in CJS modules either out of the box or with a configuration.
If you don't see such a path as a viable option forward - can you please explain (to use less knowledgeable folks) why you think that and what you believe a path forward is that would be the least surprising and easiest for users to use?
Spec compliance as well as communicating concerns to TC39 and collaborating with the committee in good faith.
Right now pre-processing / producing a static list of names is not seen as a problem in previous discussions.
Out of order evaluation was seen as problematic, but is not technically banned due to underspecification. Last meeting makes me think spec would probably lead us to ban it somehow if people sought to use it though.
Late binding could be a way of having spec compliance if the specification was changed to allow it. Doing so would require extending AbstractModuleRecord in some way and would need to go through TC39.
Giving users the ability to solve the discussed use case brought in the use case meeting - namely to be able to use named imports in CJS modules either out of the box or with a configuration.
Without a preprocessing step (all current discussions use some tool I think? idk if that matters) only late binding can provide named imports. This is what esm is doing. This approach means some compromise on static analysis since modules that could be using this feature do not have the shape guarantees of other modules. Still needs to go to TC39 as stated above probably.
With a preprocessing step, static lists can be done in spec semantics today; but, as I pointed out work on this front was never finished/was dropped due to social pressures.
Both these suffer compromise on how module.exports behaves for live binding synchronization timing and behavioral differences of Module Namespace Objects and regular Objects.
I think the right approach varies on what uses you are seeking to preserve. Neither acts 100% like babel or typescript's output. People can continue to use tooling and tooling can output new code either way though to provide different interop even if we ship something.
Last meeting makes me think spec would probably lead us to ban it somehow if people sought to use it though.
Then do so and get it banned. It's either in the spec or it's not, tbh. This "not in the spec, but still not per spec" area is wonky - the spec is intended to be a source of truth, a reference; not the people who discuss it (no offense). It's either a spec bug and it should be fixed, or its not, and the space is available for use.
Doing so would require extending AbstractModuleRecord in some way and would need to go through TC39
Alternatively to extending AbstractModuleRecord, more appropriate, I think, would be a type that sits above AbstractModuleRecord and provides the normal stage calls (instantiate, evaluate, resolveExport) plus lower-level components than GetExportedNames (like, HasName and MergeNames operations which can be used by the graph walk), which AbstractModuleRecord can then implement using GetExportedNames. In this way AbstractModuleRecord becomes the base for statically analyzable modules (and doesn't actually change much), while other modules which cannot be analyzed don't share that base. It is worth noting that you also need to introduce exceptions into the static semantics of SourceTextModuleRecords, along the lines of "if the module record is an AbstractModuleRecord ..." to skip the static checks if the imported module isn't able to be static. Its feels required, since without static names upfront, things like static "ambiguous" on conflicting reexported bindings can't be known statically. If you wanted to go that route.
I have historically been a proponent of transparent interop (see https://github.com/benjamn/reify), but I'm interested in exploring the alternative(s) with an open mind.
In that spirit, if we put our eggs in the import.meta.require basket, I imagine lots of ESM wrapper modules will be written in this style:
const exp = import.meta.require("./index.js");
export default exp;
export const { x, y, z } = exp;This simplistic style has the advantage of providing all the metadata the module system needs to instantiate the ESM version of the module (a good thing, compared to inventing some new kind of metadata), but of course it does not support live bindings, and potentially lets the contents of the default export object exp differ from the values of the named exports.
A package author who really cares about liveness could choose to write the ESM wrapper in a way that keeps x, y, and/or z updated over time, but that's not easy. Keeping exported values updated without using native live bindings is a big part of what @jdalton's esm loader does, and it requires a compiler and a small runtime library.
If we could develop patterns that make writing these ESM wrapper modules easier for human developers, then Node wouldn't have to guess the shape of CJS modules (because the wrapper would dictate the shape), and transparent interop would lose some of its appeal, which would be a good thing, I think, because it would simplify Node's ESM implementation, and it would open the door to writing similar ESM wrapper modules around any kind of non-ESM module (not just CJS).
As an example of what I mean, here's a somewhat radical idea:
What if we (via TC39) relaxed the current
export { a, b as c }syntax so that you could put an arbitrary expression in place of the b identifier? If the expression was reevaluated whenever c was used elsewhere (in the same way the latest value of b is used today), then the ESM wrapper author could write
const exp = import.meta.require("./index.js");
export default exp;
export {
exp.x as x,
exp.y as y,
exp.z as z,
};and get all the benefits of statically self-documenting exports with full support for live bindings. Because this description of the shape of the module is static, it's fixed before the index.js module is ever evaluated. You can't just blindly re-export everything that index.js exported, but I would argue that's a virtue (for static analyzability) rather than a shortcoming.
Maybe you're not sold on this particular solution, but I think we need some new ideas to move this discussion forward. And since we've been talking about "going to TC39" anyway, why not identify exactly what we want (even if it requires spec changes), and push for that?
@bmeck what if live bindings don't work with CJS imports?
Is that actually required by any of the use cases we've discussed?
Then do so and get it banned. It's either in the spec or it's not, tbh. This "not in the spec, but still not per spec" area is wonky - the spec is intended to be a source of truth, a reference; not the people who discuss it (no offense). It's either a spec bug and it should be fixed, or its not, and the space is available for use.
Spec always needs more people wanting to write up the exact semantics and that have time. This one is particularly hard, but would be glad to help you if you want. We have plenty of outstanding work to do as well in other areas for the spec and are only human.
It's either a spec bug and it should be fixed, or its not, and the space is available for use.
@weswigham it's not "the spec, or not the spec" - it's also a good-faith collaboration between implementors and the spec committee. If you think that this needs to be in the spec, then by all means, please pursue fixing the spec bug - but in the absence of a fix, it's still a bug, and it'd still be in bad faith to attempt to implement what is a known but unintentional spec omission.
@ljharb I can't find a corresponding spec bug. Should one exist?
@weswigham Nobody's filed one yet; the consensus in the meeting notes was that we didn't need to change anything but would in the future if needed. If you think it's worth filing an issue, please do - it would be most helpful to suggest spec text, and reference this issue and that notes item.
@ljharb I don't think it's a spec bug - to me it certainly feels like an affordance for non-esm modules in the graph (init in a native module in node wouldn't really be possible without it) - why would I file an issue? I'm literally one of the people advocating we utilize the feature, because it doesn't seem like a bug. I'm quite possibly one of the worst people to file such a bug.
didn't need to change anything but would in the future if needed
Comments like
I feel from the room we have a consensus of wanting something in spec but not knowing enough about the problem.
lead me to believe it was punted not just for difficulty but for lack of consensus on the issue. If you feel strongly that it is a bug (and your comments in the meeting would lead me to believe that), then perhaps someone who got the background from being at the meeting should probably at least open the spec issue - I may even contribute some candidate changes and their potential downsides (hopefully leading to why I think it's unspecified because it's unspecifiable, unenforceable, and/or undesirable); I'm not looking to champion something about enforcing a view I'm looking to challenge. As it stands, nothing in the spec forbids it, and there is no formal guidance on the matter behind a sentence in some meeting notes stating that more work is needed and the opinions of a pair of the people in attendance. There's a formal process and a spec literally to formalize issues like this, so external stakeholders (such as myself) can get visibility on the process. Like I stated a bunch of comments up; nobody can point me to a spec point that would forbid it, or even to a spec bug stating intent to specify the behavior. There's no tracability for the matter. That's what's unsatisfying here. Everyone mentions "the spec", but not backing docs other than varying opinions. The module loader is hilariously unspecified (I would have loved a fully specified loader with no host-specified behavior) because the intent is for a lot of it to be host-specific behavior (that's why there's an AbstractModuleRecord in the first place).
I'm all for deciding that module evaluation order across module type bounds is super important for node and that we need to keep it for us (conscious of what tradeoffs are being made); but that's a decision for the group to come to, and as it stands currently isn't even something at odds with the spec as it exists, so discounting that option on that basis at the very least feels disingenuous. That's why I'm frustrated.
@weswigham I explicitly pointed you to where Abstract Module Record evaluation is to occur in the spec ( https://tc39.github.io/ecma262/#sec-innermoduleevaluation ) with regards to being loaded by a Source Text Module Record in #81 (comment) .
@weswigham the authors of that spec text and the rest of the members of the committee strictly did not intend to provide any such affordance, so if you're interpreting one, then it's definitely a spec bug. tc39/ecma262#1195 is now filed, and one way or another, the spec will end up forbidding it, so pursuing that avenue of discussion isn't productive.
@bmeck It doesn't do any validation for non-sourcetext modules:
This abstract operation performs the following steps:
If module is not a Source Text Module Record, then
Perform ? module.Evaluate().
Return index.
there's no validation at all there. it doesn't even check [[Status]], because abstract modules don't have [[Status]]. And as I said the semantics of calling Evaluate on an abstract record are unspecifiable, the only requirement being that it's idempotent. Pre-executed and cached is idempotent. As I said "Evaluate" is a misleading name (given the idempotency), it's more like "GetNamespace".
as I pointed out work on this front was never finished/was dropped due to social pressures.
I missed that - is there any way the foundation could help you with that? Is that something we can talk about in the collab summit (if you're attending) to better understand how we can help create a safer environment for contribution?
I’d like to take a crack at a proposal. I’m sure this presents all sorts of issues, which I welcome people to point out, but I just want to throw some ideas out there in case any of them prove to be useful.
I like the idea of using package.json to define whether the files in a package should be treated as CommonJS or ESM. Let’s say an app I’m working on has its package.json file configured to tell Node to treat my app as ESM, and I’m using import and export statements.
I also have, oh, 100 direct dependencies for my app, like most Node apps tend to. I have no idea which of them support importing as ESM and which need to be imported as CommonJS, nor do I really want to keep track. Some presumably support either, and have separate entry points for each. This would all be defined in the package.json files for each package, and packages lacking this information would be treated as CommonJS.
In my app, I have code like this:
import { a } from 'esm-dependency';
import { b } from 'commonjs-dependency';What happens in the first line is entirely according to the ESM spec. No surprises there.
Here’s where things get interesting: What happens in the second line is that Node essentially transpiles the import statement into its CommonJS equivalent, something like:
const { b } = require('commonjs-dependency');And then we simply use the existing CommonJS machinery to import CommonJS modules.
This essentially bypasses the entire debate about how the ESM spec should handle CommonJS imports—we just wouldn’t be using it. CommonJS modules would be imported via CommonJS require, even if the syntax the user wrote was an import statement. As a user, this is what I would expect to happen. CommonJS modules being imported by the CommonJS loader is what happens today, and is unsurprising behavior for users. CommonJS modules would appear to “just work.”
There would need to be some shenanigans to avoid a global require variable appearing in ESM code, along with the rest of CommonJS’ bits; and hopefully this “transpilation” could be done in a theoretical way without necessarily needing to create new transpiled code to pass through V8. I’m sure there are other wrinkles that would need to be ironed out to take this from general theory into specific implementation.
But are there any fundamental problems to this approach? Is it something that, in general, could work?
@GeoffreyBooth one issue would be that not every use case has a package.json - one-off scripts that require each other, for example. Separately, having node “transpile” ESM syntax does in fact mean that it’s violating the spec, not bypassing it. All that said, feel free to make a proposal and everyone can review it - it’s just highly unlikely it’ll be possible (just to set expectations).
@ljharb The idea would be that within package boundaries, modules need to keep to the same module type. So my app would be all ESM, and inside commonjs-dependency everything would be CommonJS, and inside esm-dependency everything is ESM (or it exports either; not sure how that would work but that’s another problem to solve). And of course I’m not proposing solutions for all of our many use cases (--eval, etc.). This is just meant to be the kernel of an solution to potentially solve this use case. If it makes sense at least just for this, we can expand it to see if we can flesh it out to cover more or all of our use cases (probably by merging it with other proposals and implementations already out there).
The core idea is that an import of a CommonJS module should follow CommonJS semantics. Throw out all the spec requirements of ESM; that just doesn’t apply here. CommonJS is a legacy thing, and if it’s still in a project then Node imports it in this legacy mode. ESM spec applies to everything else.
@GeoffreyBooth that approach doesn't satisfy all the use cases about gradual migration within the same package/app. Regardless, the ESM spec requirements always apply when using import - something being deemed "legacy" doesn't change that.
ESM spec requirements always apply when using
import.
And I’m proposing that they don’t. In the case of CommonJS, the CommonJS implementation would apply.
That proposal would violate the spec, since the spec governs use of import, and it would set a dangerous precedent for wasm modules, HTML modules, and whatever comes in the future.
This is somewhat off-topic, however, so let's discuss further in your proposal.
Another approach would be to merge this proposal with #82 (@MylesBorins). Let’s say there’s a pluggable loader for CommonJS, and it ships with Node. You can opt into loading it somehow (code, package.json configuration, flags, whatever) or Node loads it automatically if it detects CommonJS in your dependency tree. Then instead of transpiling import x from 'commonjs-dep', such lines automatically get handled by this commonjs loader.
It’s the same outcome from the user’s perspective. The only specialness I’m adding to #82 is Node loading this loader automatically when it knows it’s needed. That will be the tricky engineering part to solve, but I’m sure we could come up with something. And the greater control the user would have over the loader would enable it to be controlled for more granular cases like gradually converting a codebase.
@GeoffreyBooth it is perfectly fine to ship something that is not ESM, but we should not label it as ESM if it violates the specification. As such, I would just leave it up to users to install some custom loader to do the behavior you describe so that we can keep the ESM label with impunity. Meteor takes an interesting approach to things and does ship customized transpilation in their main runtime and could be something to look at if this approach is desirable.
I would like to avoid proposing ideas in issue threads, I've been down that path before. Making new issue threads/PRs around different proposals would be good once use cases and features are done being talked about.
Can we reach consensus on the following?
- We agree to pursue spec compliant solutions in the nodejs/modules team and to address any non-spec-compliant changes we'd like by raising them up with TC39. We agree that it's important to do this in order to have real browser interop while supporting CJS.
- We agree that we should provide a way for users to use CJS modules in Node.js from inside ESM and that this way should be about as easy for the average user as using ESM with current userland solutions. We acknowledge this means named exports should work.
- We agree that we should discuss a compromise with regards to late bindings regardless of this issue - we should bring it up with TC39 (at the same breath as top level await which also causes late bindings if I understand correctly) and see if these concerns can be addressed at a language level.
Also - would anyone (or hopefully, two people suggesting alternatives) would be willing to bring this up for discussion in the next meeting?
@benjamingr we can make agenda items about those 3 topics; but I would not want to reach consensus in this issue on those 3 topics and treat it as accepted positions.
@bmeck I agree, I didn't mean right now in this issue just whether pursuing it is a worthwhile idea.
@benjamingr I can agree with those three points. I think there’s one issue that there’s confusion on, though: does the spec cover importing of CommonJS modules? If it doesn’t, Node is free to do whatever it wants, though obviously we should be careful to not break future things. If it does, how does it propose to solve these interoperability issues? If the spec is supposed to cover interoperability with CommonJS, but doesn’t actually say how that should be achieved, then it seems to me like the spec is incomplete.
It seems to me like some people on this thread are coming at this issue from the lens of the spec, in the sense that the spec didn’t discuss CommonJS so the challenge is basically, “how can the spec accommodate Node working with CommonJS.” But really the challenge is, “how can Node adopt the ESM spec while preserving backward compatibility with CommonJS,” which is a different thing. Node is already not fully spec-compliant, as it has CommonJS, which isn’t part of any spec. Maybe someday there will be a fully compliant Node that lacks support for CommonJS, or a compliant “mode” for Node that is so, but until then Node will operate in a hybrid form. So the question is, what shape should this hybrid form take, that offends the ESM spec the least and avoids hindering future efforts at getting to full spec compliance?
Node is already not fully spec-compliant, as it has CommonJS, which isn’t part of any spec.
Additions of APIs outside of the JS specification does not make it violate the JS spec. Performing behaviors using JS mechanism that are outside of semantics/invariants laid out of JS spec are where breakage occurs. I don't think Node is violating the JS specification by including require() as we don't change how the object model / operations / etc. work from the JS specification. We have no need to break the specification and have not currently broken it in a way that violates the basic idea of optimizations that preserve semantics are not breaking that is true of almost all compilers and VMs stating that they comply to specifications.
@GeoffreyBooth The spec potentially covers importing of anything that is importable. The fact that it's CJS, or ESM, or "third format" is irrelevant, because it still interacts with the spec-governed (and also engine-governed) JS runtime.
@GeoffreyBooth The spec potentially covers importing of anything that is importable. The fact that it's CJS, or ESM, or "third format" is irrelevant, because it still interacts with the spec-governed (and also engine-governed) JS runtime.
I'm not sure this is quite that clear - WASM will integrate at the module record level so can be a well-defined close integration with JS, while third-party loaders in NodeJS can treat defining a module as simply "returning a module namespace shell" regardless of when execution happens. We have an execute callback in the dynamicInstantiate hook, but users have no obligation to use it - they can define things that execute whenever they want, while providing named exports.
So if we can see this as a gradient from tight integration with ES modules, to more loose integration, I think the question for CommonJS here is simply where we see it on that gradient.
By definition, CommonJS does not execute like any other module record - it defers to its own loader which does its own execution. So it's more like tying two module execution trees together than the WASM integration as "just another module record".
Personally I tend to agree with @weswigham that NodeJS can interpret the spec as it wants here, and that slight execution order differences can be a very worthy tradeoff for the feature of named exports.
@guybedford any module that shares the JS realm/environment - and thus would have its effects observable - would indeed be governed by the spec. I agree that a wasm or c++ module, for example, could avoid these effects.
I do not see it as a gradient, and as has been indicated, the committee felt that whatever spec changes were needed to ensure that evaluation ordering was not altered would be made if possible. I don't think it's productive to pursue paths that are against the explicit intention of the spec, even if there is a loophole interpretation from the current wording.
@guybedford any module that shares the JS realm/environment - and thus would have its effects observable - would indeed be governed by the spec. I agree that a wasm or c++ module, for example, could avoid these effects.
To repeat the usual argument it is possible for execution to happen through the resolver, or other out-of-band mechanisms during the module pipeline processing. The modules part of the spec does not have a monopoly on all JS execution, and it's really about where you draw the line on what counts as a "module execution" from the perspective of the spec. Just because I happened to evaluate something that provides named bindings through an ES module interface, doesn't mean its execution has to be treated as an ES module, and if this is wrong then we already invalidate the spec by providing a dynamicInstantiate hook and that hook should be removed, strictly speaking, by such an argument.
The spec is designed to work for users and their needs, not as some dogma to set random walls just because Allen Wirfs-Brock happened to write the wording a certain way one weekend back in 2015.
I do not see it as a gradient, and as has been indicated, the committee felt that whatever spec changes were needed to ensure that evaluation ordering was not altered would be made if possible. I don't think it's productive to pursue paths that are against the explicit intention of the spec, even if there is a loophole interpretation from the current wording.
I'm 100% on the side of following the spec to the letter. There may well be a wording change that we need to propose to TC39 to ensure that this is not a loophole but indeed something that is supported by the specification.
There have been various discussions around this. Allowing namespaces to late-define export names was taken to TC39 and proven dead end at this point. Also previous discussion centered on cross format circular references ES / CJS problems that requires very carful execution considerations. Since we no longer have this problem anymore those previous tight concerns on execution that have been the topic of much TC39 debate over many years (remembers zebra striping!) no longer apply.
I really don't see that it will be an issue for the spec to clearly state that custom module boundaries (like the boundary between the NodeJS CommonJS loader and ES modules, which is a completely different thing to WASM boundary - to distinguish cases here, ask if both module systems can load modules from the other in a cycle) module systems may perform their own execution at whatever execution ordering they deem suitable for themselves, like the NodeJS CommonJS loader, and that their execution semantics do not need to adhere to the exact ES modules specification execution ordering at all as they are simply exposed through a module acting as a boundary, just like our dynamicInstantiate hook, where the execution callback is provided as a hint, but not a mandate.
Pursuing TC39 changes in parallel is great and I encourage and endorse it; but until those changes are moving through the process, we shouldn't let it impact our decisions here.
I understand you feel the spec is clear but you do not speak for all of TC39 - I would suggest we first seek TC39 clarification on this one more time, with an agenda item at the next meeting, and then craft a proposal from there if it is deemed necessary.
Allowing namespaces to late-define export names was taken to TC39 and proven dead end at this point
I was not aware of this, but if TC39 ruled it out, then it's dead for me too, which basically kills named exports when import-ing CJS, IMO (which begs the question of why we need transparent interop if the backwards compatibility to the "babel" way of doing things is dead).
@guybedford: do you remember the reason late-defining exports were rejected?
@giltayar it would be a large intractable spec change, due to all spec text being written around the concept of export names and export validations being known at instantiate completion. So between define the exports late, or allow earlier CJS execution, we are left with the early CJS execution option.
This is why I'm advocating CJS execution during the instantiate phase as a side-loader alongside the module pipeline.
To maintain more well-defined semantics, we could possibly only resolve CJS instantiation promises after all ES module instantiation promises have resolved. In this way, one could picture the pipeline as:
- Instantiate All (pausing on CJS execution points)
- ES Module Instantiate completion
- Start All CJS Execution triggering CJS instantate competion
- ES module execution (ES module spec execution order)
CJS modules could also still be designed to execute in tree order so that the following:
import 'cjs-1';
import 'esm-1';
import 'cjs-2';
import 'esm-2';would execute the two CJS modules exactly in spec order, during phase (3), and then the ES modules would execute in order during phase (4). The difference then just being that 'cjs-2' executes before 'esm-1', but within the respective module systems we retain post-order tree execution.
@giltayar transparent interop by default isn’t just about matching babel; imo it’s necessary with or without named imports of CJS for adoption and ecosystem migration.
Even if we disregard the specification talk for a bit; the problem with out of order execution is that you could no longer order/refactor your imports if they have side effects:
import 'a';
import 'b';Changing b from ESM to CJS would change the ordering from a, b to b, a. Which would be problematic if b relied on something from a being executed first.
Likewise, changing b from CJS to ESM would change the ordering from b, a to a, b. Which would be problematic if a relied on something from b being executed first.
These problems would be particularly hard to debug because a lot of the ordering is guaranteed and may change as you upgrade your dependencies, and we don't know the ordering just by reading the specifier in this new world either.
@bmeck yes that is the cost of the approach, that exact execution ordering only breaks down between imports of different module formats, but weighed against an entire ecosystem of expectations it seems a worthwhile one to me.
@guybedford I'm not sure how this is only a specific minority of imports, it affects any import that may change formats over time. And, when that import changes formats it will be hard to debug.
The ecosystem also has an expectation - for both require and import - that everything evaluates in lexical order. I think that’s much more important than named imports from CJS (as long as there’s at least default imports of CJS)
I've yet to see even two CommonJS packages on npm that require exact execution order between them that aren't polyfills - I have personally never written a NodeJS app that had to carefully order require statements apart from dealing with circular references.
Predictable semantics are incredibly important yes, but two-phase execution like this was the original plan for CommonJS laid out by TC39 to begin with through zebra striping. It was never deemed a spec violation then, so I'm not sure why it should be so now.
@guybedford polyfills and circular requires aren't nothing. if your polyfill is written in esm and we do out-of-band you're screwed.
If your npm package requires a polyfill to be used and is written in CommonJS, then yes, you would need your polyfill to be written in CommonJS as well.