Support `.js` extensions.
MicahZoltu opened this issue · 45 comments
import { ... } from './foo.js'Expected Behavior:
This should compile by resolving the import to ./foo.ts.
Actual Behavior:
Errors complaining that it cannot find ./foo.js.
- ES2018 module loaders do not do any extension inference.
- TypeScript compiler does not append any extension to import statements in generated files.
- TypeScript will infer
./foo.tsfrom./foo.jsduring compilation.
In order to write TypeScript that is compatible with ES2018 module loaders, you must therefore do import ... from './foo.js'. TypeScript compiler will see this import and guess that you mean foo.ts or foo.d.ts. However, ts-node does not and instead will error saying it cannot find foo.js.
While my personal opinion on this matter is that the inability for ES2018 module loaders to append a default file extension to packages that have no extension is unfortunate, it is none the less the situation we are currently in. The hope is that eventually the TypeScript compiler will be able to be given a flag that tells it to add .js when none is present, thus allowing us to write import ... from './foo' and have that generate JS like import ... from './foo.js', but that day is not yet here.
Related issues:
microsoft/TypeScript#16577
microsoft/TypeScript#28288
This sounds largely up to TypeScript to define the working functionality, then ts-node to figure out if it'll work with node.js natively. Specifying ./foo.js definitely won't work natively since that file doesn't exist, it could be named ./foo.jsx, ./foo.tsx, ./foo.ts, etc - resolution would need to be taken on with this module. It'd be a lot quicker during compilation if you specified ./foo.ts and used the "module specifier rewrite" suggestion for tsc, and in ts-node we just discard that rewrite for things to continue working as expected.
The current working functionality of tsc does not match ts-node. The links I provided discuss potential future functionality, but at the moment tsc and ts-node do not behave the same, and if tsc doesn't change its behavior then ts-node will, over time, become more and more incompatible with libraries and projects that are targeting modern ES. I see you marked this as an "enhancement" rather than a "bug", this conflicts with my understanding that ts-node and tsc are supposed to behave as close to the same as possible (which, in this case, they do not)?
I was under the impression that ts-node was using ts-server under the hood, which already does this sort of path resolution. Perhaps the path resolution logic is in a higher layer that ts-node doesn't get for free?
What "module specifier rewrite" are you referring to? The one in microsoft/TypeScript#16577 that people are asking for but hasn't been implemented or approved for inclusion yet in tsc?
@MicahZoltu In which way, other than having no transpiled output, does tsc not match ts-node? The difficulty on ts-node is two-fold, 1. there's transpile-only mode that does not resolution and 2. even with resolution, TypeScript doesn't rewrite module specifiers today. This means any file that's referenced would not actually exist in ts-node without using the original filename over transpiled output filename. The workaround here may be to add transforms that rewrite it back to the resolved filename, but this is quite complex and won't work in the first case I mention earlier. Overall, hopefully the solution in TypeScript is the module specifier rewrite (e.g. microsoft/TypeScript#16577 (comment)) because it makes life a bit easier, and also makes resolution a bit quicker (e.g. direct source references would be faster than checking each combination on the compilation side, where checking each file is up to 5x - .js, .jsx, .ts, .tsx, .d.ts, etc). There's also considerations for supporting extensions that TypeScript doesn't understand, e.g. import "foo.css", so this issue feels largely in the land of TypeScript and not ts-node to resolve.
Ah, I think I understand the problem better now. The issue isn't that ts-node is failing to compile the typescript, it is that it is failing to execute the compiled code.
script.ts
import { Foo } from 'foo.js'
new Foo()foo.ts
export class Foo {}If I tsc && node script.js it will run. If I try to do ts-node script.ts it will fail because it can't find foo.js.
I now better understand (correct me if this is wrong) that this is because ts-node doesn't pre-compile the whole project, it compiles files as-needed. So after it compiles script.ts (which will compile fine because TSC is loose with the extensions during resolution), it will begin executing it and upon reaching the attempt to import foo.js it will look for, and fail to find, a file named foo.js.
Pretty much, yes. Technically, when type-checking, it has compiled the file already but then node.js takes over resolution and will fail to find the file because .js was never written to disk. It might actually be possible for the approach you describe to work if we add a mapping layer to cache the output path from input path, but it won't work with --transpile-only so I'd prefer to wait to see how TypeScript solves this first. Or add it behind a flag if you'd really like to see it work. Alternatively, doing import {} from './foo.ts' should work, but won't work in regular node.js - it's a little chicken and egg issue that's easily solved if TypeScript does some path munging for us.
add it behind a flag if you'd really like to see it work
Ran into this again today, I would like to throw in a vote on my own issue (which I had forgotten I created until I searched GitHub) that ts-node get something like this added (behind a flag is fine). I'm working a lot with esNext code, and at the moment I'm in a pretty bad spot because if I write a library such that it works with esnext, I can't run it with ts-node which is how I do all of my testing. If I write it so it works with ts-node, then it will fail when it comes time to run in the browser.
If there isn't interest in getting a flag added to ts-node that tells it to "infer .ts if .js is missing", then I'll probably have to switch over to running my tests by compiling to JS first, then executing them (which greatly complicates my build process unfortunately).
I’m not sure I understand the request of inferring TS when JS is missing, or the linked commit. Both these describe how the module already works today. It doesn’t work with an explicit extension which is different.
foo.ts
class Foo() {}bar.ts
import { Foo } from 'foo.js' // A
import { Foo } from 'foo' // BA will compile and the emitted output will run natively in a browser. It will not work in ts-node.
B will compile and the emitted output will not run natively in a browser. It will work in ts-node.
The request here is to make A work in ts-node (perhaps with a flag) because there is no movement on fixing B in browser or TypeScript, and if I have to choose between ts-node support and browser support, I'll choose browser support. I don't want to have to make that choice though since I really like things working in ts-node (makes testing much cleaner/easier/better).
I see, I was confused by the commit referenced because I thought you meant the extension missing, not the file. You should submit a PR if you want any progress on this issue.
I took a gander at ts-node to see how difficult a PR would be. IIUC, once a single file is transpiled, it is handed off to nodejs to execute. Nodejs will encounter a line like ... require('./foo.js') and attempt to execute that. Since ts-node has only registered itself to handle .ts files, nodejs will not callback into ts-node and instead try to load ./foo.js itself. Upon failing to find the file, it will throw an exception which will halt processing.
Does this sound correct to you? How is it that ts-node receives an opportunity to handle extensionless files, like if require('./foo') was executed?
If the above is correct, then I think the way to implement this would be to register js as an extension that ts-node handles and when such a file is compiled, ts-node will either pass through to the original js handler if the JS file exists at the specified path, or it will try to compile and return a TS file if one exists at the same path, but with a .ts extension.
Does that approach seem reasonable to you?
If there’s no JS file on disk, node.js will not trigger the JS extension handler, so that doesn’t work. The only way to do this would be to rewrite paths before node.js executes the file to require dependencies.
This is getting a bit off topic, but how is it that ts-node receives a callback when nodejs encounters require('./foo')? No file exists on disk with the name ./foo, and you can't register an extension handler for '' I don't believe, and even if you could I would expect nodejs to fail during path resolution time, which is before extension handler time.
Try https://nodejs.org/dist/latest-v12.x/docs/api/modules.html#modules_all_together. .js, .json and .node are the default file extensions registered by node, but you can (minus ESM support) register any others you prefer.
Thanks for the link. I read over the flow and unfortunately it doesn't mention require extension points so it is still unclear why ts-node doesn't get an opportunity to reroute module loading for non-existent JS files. I think at this point I'm convinced that fixing this in ts-node is very non-trivial, so my inquire now is just professional curiosity.
why ts-node doesn't get an opportunity to reroute module loading for non-existent JS files
Node.js doesn't invoke any loader for a file that doesn't exist. It just enumerates require.extensions to find the file (e.g. extension-less) or assumes .js if an exact import. None of this encompasses finding a file that doesn't exist at all. You'd need to rewrite imports before node tries to resolve it for this.
If I have import { ... } from './foo' in my TypeScript file, that will emit require('./foo') I believe. ./foo doesn't exist on disk (only ./foo.ts does), yet the ts-node loader is invoked for that file I believe? This is the piece I'm struggling to follow, why is extensionless special? Or is it that "anything other than .js/.node goes through any attached loaders"?
At this point, I'd recommend you just play with it yourself by editing require.extensions and just doing console.log. There's something either in the node.js documentation or what I've said that doesn't make sense to you and I'm not sure which it is.
I think the thing you're asking is described by:
LOAD_AS_FILE(X)
- If X is a file, load X as JavaScript text. STOP
- If X.js is a file, load X.js as JavaScript text. STOP
- If X.json is a file, parse X.json to a JavaScript Object. STOP
- If X.node is a file, load X.node as binary addon. STOP
Just replace .js, .json and .node with everything in require.extensions (which includes .ts once ts-node is loaded).
Ah, I see. What I was missing was the fact that adding an entry to require.extensions adds more tests to the LOAD_AS_FILE section. So if there is a require('./foo.js'), it will check to see if ./foo.js exists, then ./foo.js.js, ./foo.js.json, ./foo.js.node, and ./foo.js.ts. Since none of those exist, it will go on to check some other places and then eventually fail without ever calling into any module loaders.
Thanks for explaining it!
Note to anyone who ends up here while trying to deal with this issue while we wait for an official fix from Microsoft: You can use a simple transformer I wrote to have the TypeScript compiler add the .js extension when emitting es2015 modules: https://github.com/Zoltu/typescript-transformer-append-js-extension
Note to anyone who ends up here while trying to deal with this issue while we wait for an official fix from Microsoft
There's no indication that anything is "broken" on the TypeScript end, or that there will be a change. Since it's completely valid, and preferable, in TypeScript to include the .js extension in imports, this seems like a ts-node bug that should not be closed.
As of ES2015, TypeScript is not emitting valid JavaScript that can execute in a browser. IMO, this is a bug in TypeScript since one of its design goals is the ability to compile to JS that runs in a browser.
What's not valid JavaScript? If you include the .js extensions, everything works great: https://github.com/Polymer/lit-html/blob/master/src/lit-html.ts#L33
The following is a valid TypeScript file:
import { Foo } from './foo'
new Foo()The following is the generated JavaScript if you use es2015 modules and target es2018:
import { Foo } from './foo'
new Foo()The latter will not execute in a browser, and there are no plans to change the ES specification such that the latter will execute in a browser. Either TypeScript should be changed such that the above TypeScript file is flagged as invalid (meaning, TypeScript throws a compiler error because you left off the extension) or the TypeScript compiler should be changed such that it appends the proper extension during emit.
Node.js doesn't invoke any loader for a file that doesn't exist
Yeah, but we're not talking about Node invoking require hooks for non-existent .js files, we're talking about Node invoking hooks for .ts files (that contain code like import foo from './foo.js').
So, Node.js will invoke ts-node's hook for .ts files, and when it does, ts-node can re-write/handle the .js specifiers any way it wishes, to make things work.
@trusktr Feel free to submit a PR if the limitations don’t seem like an issue. I am always happy to accept a PR improving functionality.
Would doing something similar to the following at the top of the getOutput functions work?
code = code.replace(/(?<=from\s*'\S*)\.js'$/, "'")
It removes the .js from the specifiers. It would need to cover more corner cases. For most cases that would work I think, and would be behind an option.
@trusktr Wouldn't it be better to implement this via a TypeScript transform? Is that code running on TypeScript or JavaScript output?
That could run on the code before it gets transpiled by ts-node. Does ts-node support transforms? Maybe that's another way too.
I think probably the simplest thing is for an end user to configure ts-node to use https://github.com/Zoltu/typescript-transformer-append-js-extension, and to document this in the README or somewhere.
The docs aren't clear on how to use custom transformers.
deleted
In case anyone stumbles here,
The transformers option to register() is of the CustomTransformers type:
interface CustomTransformers {
/** Custom transformers to evaluate before built-in .js transformations. */
before?: (TransformerFactory<SourceFile> | CustomTransformerFactory)[];
/** Custom transformers to evaluate after built-in .js transformations. */
after?: (TransformerFactory<SourceFile> | CustomTransformerFactory)[];
/** Custom transformers to evaluate after built-in .d.ts transformations. */
afterDeclarations?: (TransformerFactory<Bundle | SourceFile> | CustomTransformerFactory)[];
}I deleted the above, that transform is opposite of what I was wanting, which is to remove the .js extensions, not add them.
Ok, here's how to add a transform to ts-node as an end user. This one is opposite of @MicahZoltu's, it removes the .js extensions:
// first write a transform (or import it from somewhere)
const transformer = (_) => (transformationContext) => (sourceFile) => {
function visitNode(node) {
if (shouldMutateModuleSpecifier(node)) {
if (typescript.isImportDeclaration(node)) {
const newModuleSpecifier = typescript.createLiteral(node.moduleSpecifier.text.replace(/\.js$/, ''))
return typescript.updateImportDeclaration(node, node.decorators, node.modifiers, node.importClause, newModuleSpecifier)
} else if (typescript.isExportDeclaration(node)) {
const newModuleSpecifier = typescript.createLiteral(node.moduleSpecifier.text.replace(/\.js$/, ''))
return typescript.updateExportDeclaration(node, node.decorators, node.modifiers, node.exportClause, newModuleSpecifier)
}
}
return typescript.visitEachChild(node, visitNode, transformationContext)
}
function shouldMutateModuleSpecifier(node) {
if (!typescript.isImportDeclaration(node) && !typescript.isExportDeclaration(node)) return false
if (node.moduleSpecifier === undefined) return false
// only when module specifier is valid
if (!typescript.isStringLiteral(node.moduleSpecifier)) return false
// only when path is relative
if (!node.moduleSpecifier.text.startsWith('./') && !node.moduleSpecifier.text.startsWith('../')) return false
// only when module specifier has a .js extension
if (path.extname(node.moduleSpecifier.text) !== '.js') return false
return true
}
return typescript.visitNode(sourceFile, visitNode)
}
require('ts-node').register({
// ... other options ...
// then give the transform to ts-node
transformers: {
before: [transformer]
},
})Interesting approach @trusktr. I'm assuming you add .js to all of your import statements, and then the idea is to use this transformer so that you can run those in ts-node?
@MicahZoltu Yeah, or in Webpack apps.
FWIW, another option is to install a service worker to make the browser automatically add back the .js. I agree with @justinfagnani that the correct thing to do here is to make ts-node handle .js extensions the same way every other TS tool does, but until that's the case, this seems like the smoothest workaround (I had previously been using @MicahZoltu's plugin to do this on the TS side, but it has a poor interaction with tsc --watch where it seems to not run consistently, though this could be an artifact of my flycheck setup running two copies of the latter).
self.addEventListener('fetch', (event) => {
if (!event.request.endsWith('.js')) {
event.respondWith(fetch(event.request));
return;
}
event.respondWith(async () => {
const resp = await fetch(event.request);
let text = await resp.text();
text = text.replace(/^(import[^;]*from\s*')([^']*)(';)/g,
(full, prefix, path, suffix) => {
if (path.endsWith('.js')) return full;
return `${prefix}${path}.js${suffix}`;
});
return new Response(text);
});
});
The downside is that you need to wait until after the service worker loads to start the initial module load.
EDIT: This just doesn't work reliably enough. This is really a problem and I would argue it's on ts-node to fix, since it's the one piece of tooling that is inconsistent with TypeScript's and everything else's behavior here.
@shicks The transformer should work fine with ttsc --watch. I use it in projects that do watching and it works. My guess is that somewhere in your infrastructure you are using tsc instead of ttsc or some other mechanism for compiling that doesn't apply the transformer.
That's probably the case - I completely forgot about ttsc being a thing, so probably my flycheck is running ordinary tsc and they're racing.
In any case, I ended up just wrapping ts-node and esm into my own loader for testing so that I can keep my sources pristine and only white-glove what ts-node sees for testing. @trusktr's solution mostly worked, except I needed to unwrap the outermost arrow function to make it actually run anything rather than just crashing.
Unfortunately ttsc does not integrate nicely with WebStorm so it would still be nice if this would be fixed...
Just a thought:
If TS is already performing module resolutions during typechecking, can we cache those results and use them when require() calls happen? If this approach works cleanly -- I'm not 100% convinced it will -- then it lets us piggy-back on work already being done by the compiler, and we conveniently get the exact same behavior. (Mapping from .js paths to .ts source, "paths" mapping)
Potential problems:
- A file that uses static
importand dynamicrequire. If both request strings are the same, then static import's resolution is used for dynamic require. (might not be desired) If request strings are different,require()might not behave the way you want. - If TS resolves anything to a
.d.tsfile when using composite projects. That's not what we want.
Given that Microsoft has put their foot down (closing and locking microsoft/TypeScript#16577) and will not change course on this, could we consider reopening this to fix it in ts-node? I'd rather not be forced to use ttsc just to allow a smoother experience in nodejs.
A few thoughts:
ts-node's ESM loader already resolves .js to .ts. This is because our ESM loader hooks node's built-in ESM resolver, adding .js -> .ts resolution. node --loader ts-node/esm
ttsc can be used in ts-node via the following tsconfig:
{
"ts-node": {
"compiler": "ttypescript"
},
// configure transformers as normal for ttsc
There are 2x possible ways to implement .js->.ts resolution:
- compile-time transformer: import statements are rewritten, node's resolver is untouched.
- runtime resolver hook: import statements are compiled normally, node's resolver logic is extended to resolve
.js->.ts
The latter matches what already happens via --loader ts-node/esm and may play nice with a built-in tsconfig-paths or project references resolver.
The latter also supports dynamic module loading, where the compiler does not have an import statement to rewrite. For example require('./plugins/' + pluginName + '.js') and we want node's runtime resolver to resolve this to a .ts file.
@cspotcode Running node --loader ts-node/esm node_modules/mocha/bin/_mocha test/**/*.test.ts throws:
Unknown file extension "" for .../node_modules/mocha/bin/_mocha
Is that expected?
Yeah, it's a node bug. I think if you search around you'll find the tickets and comments; I know I've explained it a few times before both here and on node's issue tracker.
Reference: mochajs/mocha#4267
I think I'm hitting a problem like this but its because I'm using BullMQ to create a Node Worker thread for my BullMQ worker
it requires a file path as input for the worker implementation which is just a default function. ts-node does not appear to compile the .ts file I have for this worker as its not used anywhere. Attempts to invoke it in the .ts file do not appear to help either.
I cannot load a .ts file unless I migrate from CommonJS to ESM modules. Jest does not fully support ESM modules yet and all my existing unit test break when I do that.
I think my best bet for now will be to move off of ts-node and onto using tsc watch or something.
FWIW, I've found that using TS and Node together is just more trouble than it's worth. There's some rumblings about native support coming at some point for at least a subset of TS syntax, but overall I've had much better success with VMs that can support full TS directly (e.g. deno and bun), and then using tsc with the noEmit setting to get proper typechecking.