ts-node with project references in a lerna monorepo
kerdany opened this issue ยท 37 comments
I'm using lerna
with typescript project references for a node application containing two packages. Package lib
and package app
(which depends on lib
).
I've configured typescript project references so that package lib
is built automatically whenever we run tsc --build
inside package app
(i.e. to build for production).
Here's how the tsconfig.json
files are configured for each of the packages:
packages/lib
/tsconfig.json:
{
"compilerOptions": {
"strict": true,
"rootDir": "src",
"outDir": "dist",
"tsBuildInfoFile": "dist/.tsbuildinfo",
"composite": true
}
}
packages/app
/tsconfig.json:
{
"compilerOptions": {
"strict": true,
"rootDir": "src",
"outDir": "dist",
"tsBuildInfoFile": "dist/.tsbuildinfo",
"incremental": true
},
"references": [
{ "path": "../lib" }
]
}
Currently, running tsc --build
(inside app
) compiles typescript to javascript in the dist
directory in each of app
and lib
perfectly fine, the setup runs flawlessly in production mode.
The problem, however, is that trying to run the project in development with ts-node
via nodemon --exec 'ts-node src/index.ts'
in development fires the following error:
/path/to/packages/app/node_modules/ts-node/src/index.ts:245
return new TSError(diagnosticText, diagnosticCodes)
^
TSError: โจฏ Unable to compile TypeScript:
src/index.ts(1,23): error TS2307: Cannot find module '@myproj/lib'.
...
What seems to be happening, is that ts-node
is looking for the @myproj/lib
package inside node_modules
directory (symlinked by lerna
), instead of compiling it on the fly through typetcript's project references, as is setup inside both tsconfig.json
files.
I validated my theory by:
- Running a regular
tsc --build
first (which also builds thelib/dist
code). - Then running
nodemon --exec 'ts-node src/index.ts'
again, and it ran fine then.
Which means that ts-node
in this case is loading lib
via the compiled .js
code inside lib/dist
(symlinked by lerna
), NOT via compiling its .ts
code on the fly (via references).
I'm using ts-node@8.4.1
(currently the latest version).
Some questions:
-
Doesn't
ts-node
currently support project-references yet or passing the--build
flag totsc
yet? -
Am I doing something wrong in my (e.g.
tsconfig.json
) configurations that's causingts-node
not to compile/buildlib
. -
I can see that a new
--build
flag is being added tots-node
in themaster
branch (commit), but it seems that it's irrelevant totsc
's--build
flag.
Note: Currently I'm working around this (without ts-node) via nodemon --exec 'tsc --build && node dist/index.js'
till I get this figured out.
Thanks!
- I don't think so, though I can replicate project references working for other cases. It's using the language services API today which has typically had no problems, but I'd have to ask if you can build a simple repro for me to work with on this to confirm.
- I don't think so, more likely that references don't work as expected.
- Correct, I can probably rename to
--emit
to clarify the use-case. However, I run into issues with project references on the latestmaster
build that need to be fixed (so your example might help me too) ๐ฆ
Thanks @blakeembrey
- Sure, I will prepare a sample repo to reproduce the issue first thing in the morning (UTC+2 here).
Thanks!
Hi again,
I've created code to reproduce issue here:
https://github.com/kerdany/ts-node_issue-897
@kerdany Thanks! Interestingly I get the same error in development with VS Code and tsc
, until I enable --build
. This makes sense, since I assume that TypeScript is actually building references
first when you specify --build
. It does make it trickier for ts-node
though, I guess it should also build those dependencies somehow. We might have to do something in memory to make this possible...
Another interesting part of this is that it's related to node
and TypeScript's understanding of module resolution. You can actually make this work with the current release of ts-node
by importing directly the other .ts
file:
import { greet } from "@myproj/lib/src/index";
Edit: Don't expect this to work in the next major version though, I've been having a lot of trouble with getting project references working and this also fails with VS Code tsc
.
Have you done any progress or got any clues for this? I just ran into the same issue as well
I also ran into this trying out packages in yarn workspaces. When wanting to debug with node -r ts-node/register
as a work around for now I am composing a root tsconfig.json with all relevant references
and running a tsc --build --watch
running separately. Would be great to have project references resolve and build with ts-node alone.
FYI it looks like they've added API support for building project references.
interface SolutionBuilder<T extends BuilderProgram> {
build(project?: string, cancellationToken?: CancellationToken): ExitStatus;
clean(project?: string): ExitStatus;
buildReferences(project: string, cancellationToken?: CancellationToken): ExitStatus;
cleanReferences(project?: string): ExitStatus;
getNextInvalidatedProject(cancellationToken?: CancellationToken): InvalidatedProject<T> | undefined;
}
FYI: the obvious workaround is to set TS_NODE_TRANSPILE_ONLY=true
but then you won't get any type checks.
@krzkaczor, I tried setting this on the command line (using --transpile-only
and -T
) and it makes no difference -- project references still aren't built.
I ended up using concurrently
to build the project references for all the libraries within our monorepo (all inside a lib
directory) like so:
"scripts": {
"start": "concurrently \"tsc -b -w ../lib\" \"nodemon -w ./src -w ../lib --ignore 'lib/*/src/*' -e js,jsx,gql,ts,tsx --exec ts-node src/server.ts\"",
}
Here, the lib
directory contained a tsconfig.json
with references to all the composite projects within the directory.
I'd also like to use ts-node with project references. Until this is supported, I'm using the following workaround:
tsc-watch -b --onSuccess 'node dist/index.js'
On my project, this picks up changes slightly faster than using concurrently
with tsc
and nodemon
.
Any progress on it?
Line 921 in f77e1b1
This comment is confusing me. Does it mean to say that there is a --build
flag in ts-node? I can't find it in the code or the documentation. I assume it's just an artifact of a work in progress? (I'm trying to use naked pnpm rather than lerna, but I don't think that makes much difference.)
Here is a ~hidden pending change to the typescript compiler API wiki: https://github.com/microsoft/TypeScript-wiki/pull/225/files#diff-709351cd55688fbcb7ec0fc9973ee746R407
I am trying to figure out adding project references support to ts-node and then ts-node-dev, but I have low confidence that I can figure it out as a beginner contribution. If any of the maintainers have explored this before, can they please share their findings so far?
My basic guess is that we need a new CLI flag (--solution
/--S
) that basically replaces createIncrementalCompilerHost
with createSolutionBuilderHost
here:
Lines 787 to 799 in f77e1b1
If it were that simple, it'd probably be done by now. I think that SolutionBuilderHost
used to extend CompilerHost
, but no longer does. CompilerHost
rolls up to ModuleResolutionHost
whereas SolutionBuilderHost
rolls up to ProgramHost
.
BTW, where is it that the compiled in-memory script is executed? I don't have any ideas of what to do after ts.createSolutionBuilder(host, Array.from(rootFileNames), {}).build();
Seems like I need to get a Program
out of there somehow?
@sheetalkamat might know exactly what to do.
Line 921 in f77e1b1
This comment is confusing me. Does it mean to say that there is a
--build
flag in ts-node? I can't find it in the code or the documentation. I assume it's just an artifact of a work in progress? (I'm trying to use naked pnpm rather than lerna, but I don't think that makes much difference.)
--build
is referring to the incremental
flag in tsconfig.json:
if(options.emit && config.options.incremental)
options.emit
refers to ts-node's --emit
flag/option, and config.options.incremental
refers to TypeScript's incremental
tsconfig option.
Here is a ~hidden pending change to the typescript compiler API wiki: https://github.com/microsoft/TypeScript-wiki/pull/225/files#diff-709351cd55688fbcb7ec0fc9973ee746R407
I am trying to figure out adding project references support to ts-node and then ts-node-dev, but I have low confidence that I can figure it out as a beginner contribution. If any of the maintainers have explored this before, can they please share their findings so far?
My basic guess is that we need a new CLI flag (
--solution
/--S
) that basically replacescreateIncrementalCompilerHost
withcreateSolutionBuilderHost
here:
I think we can get away with enabling this behavior by default, or at least enabling by default when --files
is enabled. We might need to make --files
default behavior in the future anyway.
BTW, where is it that the compiled in-memory script is executed? I don't have any ideas of what to do after
ts.createSolutionBuilder(host, Array.from(rootFileNames), {}).build();
Seems like I need to get aProgram
out of there somehow?
Here is where we hook into node's module loading mechanism:
Lines 1022 to 1049 in f77e1b1
Node internally reads the file's contents from disk, then passes it to _compile
. We wrap the _compile
function to take the source text, compile it, and pass the emitted output to node's native _compile
implementation, which handles module execution.
Isn't typescript's --build
CLI flag more related to composite: true
and project references (v3.0) while incremental
and tsBuildInfo
(v3.4) can be used regardless of whether there's a typescript project-reference involved or not?
https://www.typescriptlang.org/docs/handbook/project-references.html#caveats-for-project-references says:
to preserve compatibility with existing build workflows,
tsc
will not automatically build dependencies unless invoked with the--build
switch.
which I don't really understand - but why wouldn't ts-node need to respect this opt-in behavior?
Has this been reattempted since the API was made public to the extent that it is currently, and with the knowledge of the documentation in the wiki PR? I didn't see any branches here for it. Just wondering what trees the community can avoid barking up.
The following answers are based on memory. They might be wrong, they're not perfectly written, and they're long, but I tried to provide as much detail as possible.
Isn't typescript's --build CLI flag more related to composite: true and project references (v3.0) while incremental and tsBuildInfo (v3.4) can be used regardless of whether there's a typescript project-reference involved or not?
Correct, --build
/ composite
imply and require incremental
. If I had to guess, the comment you see in our source code about --build
is conflating the two. The comment is potentially confusing; I wouldn't focus on it too much.
why wouldn't ts-node need to respect this opt-in behavior?
That's a good question. If the user runs a script in projectA which imports a file from projectB, what should we do? Should we follow project references and compile the file in projectB, using projectB's compiler options, essentially defaulting to --build
mode? Should we eagerly type-check the entirety of projectB?
Also, ts-node
has its own ignore
option which determines which files we do/don't compile. Also, is having our own ignore
option a good or bad idea?
To get more detailed, tsc
and node
load files in different ways, and we need to somehow bridge the gap.
TypeScript has the luxury of eagerly loading all projects and files, type-checking and emitting them all at once. It starts with a single tsconfig.json
. It follows all project references, globs for all "files"
/"includes"
, recursively parses all import
statements, and adds those files, too. This process pulls more and more projects and files into the compilation, and all must be compiled and emitted. tsc
's job is to eagerly pull all files into memory, parse and typecheck them all, and emit .js
for all.
node
, on the other hand, loads files on-demand when require()
d or import
ed. When this happens, ts-node
must ensure that tsc
has already looked at the file. If it hasn't, ts-node
forcibly adds it to the "files"
array, triggering tsc
to parse, type-check, and emit it. The language service is well-suited to this on-demand style of compilation, but I'm not sure on-demand fits well with project references.
I use the term tsc
loosely to refer to the various TypeScript APIs we use.
Possible simplifications
This all needs research, but here are a few things that might simplify ts-node
and help us support project references:
- make
--files
on by default and/or required for project references - Disallow "forcibly adding" files as described above. If a file is not included in your "files" or "include" array, we throw a helpful error
About incremental and tsbuildinfo
I believe that implementing project references with the performance benefits of incremental
will require writing all output into a separate, ts-node private cache. Related to #1161
ts-node
uses potentially different compiler options than tsc
, so our emitted output may look different. It can't be written to disk in the same place as tsc
's output. For example, the user might specify TS_NODE_COMPILER_OPTIONS
. Also, we override certain sourcemap options.
I realized I should clarify something:
That's a good question. If the user runs a script in projectA which imports a file from projectB, what should we do? Should we follow project references and compile the file in projectB, using projectB's compiler options, essentially defaulting to --build mode? Should we eagerly type-check the entirety of projectB?
What I mean by this is, if you import a file from projectB, then node
is going to have a require()
that needs to happen. What do we do when that file from projectB is require()
d?
@JasonKleban I've been thinking about a good place for a beginner to start contributing to ts-node
. I think #1161 is a great starting point. All contributions are appreciated, so of course you can focus your efforts anywhere you want. But if you're looking for something straightforward that will bring us closer to project references, #1161 is a great option.
Bump
I also have this problem and it would be really nice to avoid running the extra tsc --watch
.
This is a bit of a hack but I've written a small script ts-builder
that can be used with Mocha to do tsc --build
before loading the compiled JavaScript. It will emit the files as configured in tsconfig.json
. This is what I'm now using with Mocha by doing:
mocha -r @theomessin/ts-builder/register **.test.ts
Should do until ts-node
supports project references.
This thread is really hard to find... Been on the hunt for hours on this issue.
It's a really necessary feature
Do you want to try your hand at writing a pull request for it? Check out #1514 and see if there's anything you want to help with.
Alternatively, if your employer is willing to pay for the feature to be prioritized, we might be able to work something out. We've not done that sort of thing before but we could give it a try.
https://gitlab.com/darren-open-source/mo-ts-scaffold
If it helps at all, I setup a basic TS project with modular entrypoints... I've created a list of objectives (Which were based on many TS complaints across multiple forums/tickets) and have basically resolved most of them besides this issue.
Happy for anyone to check it out and try get things working. Hoping to create a good minimum viable TS (For node) project so developers have a great starting point.
I've found that, unfortunately, for any decent project there needs to be a build pipeline. I didn't go towards babel, as I prefer webpack to just put everything into one bundle file... This also solves a lot of potential deployment and production reference issues.
I've just stumbled upon this very same issue. I couldn't find any info on the best possible way to work with ts-node
inside a monorepo. Tried using both "require": ["tsconfig-paths/register"]
and "experimentalResolverFeatures": true
with no luck. :/
+1 I would also love a way of running ts-node on a cli that lives in a monorepo (and imports from other monorepo packages using tsconfig references)
Essential for developing monorepo cli apps. +1.
I have no solid solutions for this problem yet, but I'll share my progress here with my production project - https://github.com/gridaco/cli
thanks @zomars !!!
I was having issues with a custom typing and the files: true
flag solved the issue.
This is my tsconfig.json of the test folder that references the source and has to reference a custom typing the source uses:
{
"extends": "../tsconfig.json",
"compilerOptions": {
"composite": true
},
"include": ["./", "../types/index.d.ts"],
"references": [{ "path": "../src" }],
"ts-node": {
"files": true,
}
}
Hello, I was searching for a solution for like 2 weeks, and I think I found a perfect workaround. the whole idea is that I run my main app using nodemon and ts-node, and except watching my app folder, i added another folder to watch but not the src folder of my library but the build (output folder) and extended the watch to .js files as well. Then, run tsc -b -w on my library which watches the library itself, and rebuild if necessary, and if it emits new .js files, nodemon catches it as well. and it is even in the right order
Hello, I was searching for a solution for like 2 weeks, and I think I found a perfect workaround. the whole idea is that I run my main app using nodemon and ts-node, and except watching my app folder, i added another folder to watch but not the src folder of my library but the build (output folder) and extended the watch to .js files as well. Then, run tsc -b -w on my library which watches the library itself, and rebuild if necessary, and if it emits new .js files, nodemon catches it as well. and it is even in the right order
so assuming you have like 3 different libs and 1 app in your monorepo... do you start basically 4 processes each time you start coding (3 tsc -b -w
and one nodemon
)?
also this workaround means that each change in the lib would be propagated within 2 rebuilds (1 of the lib itself and 1 of the app that is using the lib), but the changes in the app would be propagated within 1 rebuild, which means inconsistency of the time interval between saving changes and ability to run the code...
it indeed does the job but not without tradeoffs
Ran into this issue today as well, although I'm just using yarn workspaces rather than lerna. Maybe worth changing the issue title since this is a broader problem? I've reproduced with a minimal setup here if it helps anyone: https://github.com/daniel-savu/ts-node-bug
I have an npm workspaces and have same issue
Anybody have any insight into this as of March 2024? Is this supported yet? Thanks!
Anybody have any insight into this as of March 2024? Is this supported yet? Thanks!
I ended up avoiding ts-node and other typesctipt runners. Here is an example repo for how achieve live-reload and so on solely with node
built-in watch mode and tsc
https://www.npmjs.com/package/tsc-watch can be used as well for tsc driven workflows