Incremental build performance dropped

Question

Incremental build performance dropped

Closed this issue 7 years ago · 57 comments

Hi,

I noticed that our incremental builds started to be considerably slower than they used to, so I ran webpack with --profile --progress and older versions of ts-loader to see if there are differences. It seems that v0.5.2 is twice as fast as the latest version. I'm unsure how to interpret the profiling data and the webpack source code is a complete mess, so it doesn't help much in that regard.

Here are the stats:

// ts-loader v0.5.2
379ms build modules   
19ms seal
39ms optimize
60ms hashing
35ms create chunk assets
2ms additional chunk assets 
0ms optimize chunk assets 
0ms optimize assets 
99ms emit
Hash: fdd096aa48ded051dda3
Version: webpack 1.12.2
Time: 639ms

// ts-loader v0.5.6
357ms build modules   
19ms seal
50ms optimize
51ms hashing
40ms create chunk assets
4ms additional chunk assets 
0ms optimize chunk assets 
826ms optimize assets
78ms emit
Hash: fdd096aa48ded051dda3
Version: webpack 1.12.2
Time: 1429ms

It looks like there is a lot of time spent in "optimize assets". I don't know what's causing this, but I have a feeling that ts-loader is using some extra dependencies now which may have the side-effect of enabling some webpack optimizations that are meant for production, not development.

Answer 1 · 2015-10-08T12:11:49.000Z

Hmm... interesting. Thanks for reporting, I'll take a look.

Answer 2 · 2015-10-08T15:09:30.000Z

I did a little profiling with node-inspector and it seems that time is spent calling languageService.getSemanticDiagnostics(). Can't figure out from the history, if it's called more often than it's supposed to be or if it's a side-effect of some other change. Any ideas?

Answer 3 · 2015-10-08T15:31:11.000Z

Could be this commit: dc53ef6

Before it was checking for "visited" modules.

EDIT: it runs it for every TS file in the project and even lib.d.ts

Answer 4 · 2015-10-08T15:44:57.000Z

Yea, that was my thought as well. It was put in place to fix this but I think #68 is the way to go and there is a decent chance that it will be more performant.

Answer 5 · 2015-10-08T15:52:28.000Z

Yes, seems a reasonable assumption. Also, in my case there were only ~20 files or so, because the rest are still ye olde' JS files, but in a larger project, this could sum up considerably to the overall incremental build time.

Answer 6 · 2015-10-08T20:35:32.000Z

We ran into this very recently. When bundling very large projects with ts-loader, it takes a very long time, which makes iterative development very difficult. tsc doesn't take nearly so long so we've even considered using a gulp task or npm script to pipe tsc first and then webpack the js files just to speed things up. If this can be solved in ts-loader we would be happy! 😀

Answer 7 · 2015-10-08T21:10:50.000Z

@use-strict I'm working on reproducing locally. I'm seeing something similar to you, but I just wanted to confirm whether or not the numbers you posted were for the initial build or for a change in watch mode. Here's some numbers that I'm seeing:

// initial build
2589ms build modules    
14ms seal
5ms optimize 
14ms hashing
19ms create chunk assets
4ms additional chunk assets 
0ms optimize chunk assets 
784ms optimize assets
7ms emit 
Hash: 58c08445fb896261fe7e
Version: webpack 1.12.2
Time: 3460ms

// after making a change while in watch mode
85ms build modules    
4ms seal 
5ms optimize 
10ms hashing
11ms create chunk assets
4ms additional chunk assets 
0ms optimize chunk assets 
805ms optimize assets
11ms emit
Hash: 1b9954af6b2aacc4d866
Version: webpack 1.12.2
Time: 1023ms

In other words, for the initial build both "build modules" and "optimize assets" takes a long time, but a change in watch mode results in a small "build modules" but still a large "optimize assets"

Answer 8 · 2015-10-08T22:35:53.000Z

Alright, so after playing with this a little bit more my hopes at it being a quick-fix are gone. I was able to get the ~800ms "optimize assets" down to ~500ms in my test. That time is spent generating errors, and I'm not sure how much more I can reduce that (but I haven't given up yet).

However, I did want to point out there are multiple scenarios & timings to take into account here. There is the "first build" vs an "incremental build" in watch mode, and there is the time to emit output for the modules and time to generate errors. As things stand right now the first build scenario performance is pretty abysmal. In my test it takes 2600ms to generate modules for the first build, but only 85ms to generate modules for an incremental build. At my job we have a project that takes around 20s to generate the first build but less than a second for incremental builds. The point here is that if you're not already using watch mode I highly suggest you do so. That said, I think there are definitely performance gains to be made for the first build and I will look deeper into that.

Regarding the time to generate errors, it is a constant for both the first build and incremental builds, but it's also not a great constant.

So to recap:

Scenario	Action	Time	Notes
First Build	Emit	2600ms	This is bad and I think it can be better
First Build	Gen. Errors	500ms	This is also bad but I'm not sure how much I can fix it
Second Build	Emit	85ms	This is working as intended
Second Build	Gen. Errors	500ms	Same as above

Answer 9 · 2015-10-08T22:37:42.000Z

Also, I wanted to point out that in my test running tsc takes about 1000ms. So if I can get the first build emit time for my test down to 500ms I would consider that a huge win.

Answer 10 · 2015-10-09T07:00:16.000Z

@jbrantly , initial build time is irrelevant. It's expected to take a long time and it's not a problem as long as I don't have to kill the watch and restart it all the time.

Now some remarks and questions:

v0.5.2 of ts-loader did not have this overhead, provided it was only checking changed files for errors.
Checking ALL of the files should scale linearly with project size, which is very bad for us (and for @SonofNun15 apparently)
Why do we need to check files that have not changed for errors? I know you probably had a good reason for that change, but I need to understand it in the hopes that I can try to provide a better solution. It's also not obvious for me from the linked test.
Wouldn't the fix with program.getSemanticDiagnostics() work?

Answer 11 · 2015-10-09T09:23:30.000Z

Checking ALL of the files should scale linearly with project size

You're probably right, but this is something I want to look into more. I don't know if maybe TS has some optimizations in this area that I'm simply missing out on because I'm doing something silly.

Why do we need to check files that have not changed for errors

Consider app.ts -> dep.ts. dep exports something which app uses, but dep changes that export in some way which makes app's usage incompatible. If we only check changed files we won't see the error since app didn't change.

Somewhat related is #52 which deals with the dependency tree. If I'm thinking about things correctly, once that's fixed in the scenario above webpack would consider dep to be a dependency of app, which means if dep changed app would get rebuilt as well. This is bad for timing emit-wise, but could potentially allow going back to the "visited" pattern for diagnostics which might be a net win.

Wouldn't the fix with program.getSemanticDiagnostics() work?

That's what I tried yesterday. It took the time in my test from 800ms to 500ms, so while it was better it was not a magical fix out of the box. But like I mentioned, it's possible I'm doing something silly elsewhere that's causing TS not to be optimized for this case.

FWIW, I'm using this for my test.

Answer 12 · 2015-10-09T09:27:59.000Z

Just brainstorming potential fixes here: It might be possible to use transpileModule for all emits while still updating the language service (or just a program). Then after compile get diagnostics like we do today. This could reduce the initial build time, but keep the same amount of time to generate diagnostics. One problem with this approach is that it could fail for some applications since it assumes that isolatedModules is turned on.

Answer 13 · 2015-10-09T09:34:52.000Z

I will investigate more today. Looks likeprogram.getSemanticDiagnostics() still iterates through program.getSourceFiles() (https://github.com/Microsoft/TypeScript/blob/master/src/compiler/program.ts#L583-L591)

Answer 14 · 2015-10-09T11:26:39.000Z

I'm still not 100% sure of how the entire process works and there is little documentation to go on, so please just go along me:

Webpack starts with an entry point and recursively includes dependencies, following require calls. At least this is what happens for regular *.js files, without any loaders coming into play.

Now if we add ts-loader, this basically means that every *.ts file spawns a ts-loader instance. Even so, ts-loader itself caches a single typescript instance per project. At this point typescript dependencies are different from webpack dependencies.

Typescript needs to follow any dependencies for type-checking, and that's including any files that are not necessarily referenced at runtime via a require. It's these filesthat webpack has no knowledge of.

After ts-loader is done with a file, a *.js file results and webpack follows again the dependencies that are require calls.

For incremental builds, webpack watches all files in the project, so detecting a change is not a problem. The problem is figuring out which files were actually changed since the last incremental build, which webpack cannot help us with because it have no knowledge of them.

So to sum things up, we need to build the list of changed files ourselves, using only information coming from the compiler API and/or filesystem, maybe using webpack just to get the paths that are being watched. And we need to do this faster than it takes to get all errors from all files in the project. Or, we need to find another way of getting the errors in a cheaper way.

Answer 15 · 2015-10-09T13:12:31.000Z

Webpack starts with an entry point and recursively includes dependencies, following require calls. At least this is what happens for regular *.js files, without any loaders coming into play.

Yes

Now if we add ts-loader, this basically means that every *.ts file spawns a ts-loader instance.

Roughly. To be pedantic the "loader" function is called for each .ts file.

Even so, ts-loader itself caches a single typescript instance per project.

Yes. (per the "instance" loader option and per compiler instance)

At this point typescript dependencies are different from webpack dependencies.
Typescript needs to follow any dependencies for type-checking, and that's including any files that are not necessarily referenced at runtime via a require. It's these filesthat webpack has no knowledge of.

Right. Simple case would be a .d.ts file which webpack never runs through the loader or attempts to bundle.

After ts-loader is done with a file, a *.js file results and webpack follows again the dependencies that are require calls.

Yup. By the time the second file is put through the loader TypeScript generally has all of the files for the project loaded. This is actually kind of a bad thing though for various reasons.

For incremental builds, webpack watches all files in the project, so detecting a change is not a problem.

Well, to be clear, webpack watches all files that were put through the loader or explicitly added as a dependency using the addDependency API. This can be (and currently is) different from what TypeScript has loaded. Right now we manually add all .d.ts files as dependencies so that they're watched by webpack.

The problem is figuring out which files were actually changed since the last incremental build, which webpack cannot help us with because it have no knowledge of them.
So to sum things up, we need to build the list of changed files ourselves, using only information coming from the compiler API and/or filesystem, maybe using webpack just to get the paths that are being watched.

I'm not sure this is a problem. Webpack can tell us which files were actually changed. For instance it does so today for .d.ts files. The problem is detecting errors in files that were not actually changed but depended on files that were. And webpack could possibly help us there if we instruct it to do so by explicitly adding the deep tree of dependencies that TypeScript knows about to each file.

In other words, instead of calling a blanket getDiagnostics on the entire program, we could potentially build a list of files that changed concatenated with a list of files that depended on those files and call getDiagnostics on those.

As an example, suppose app.ts -> dep.ts.

If dep.ts changes, we need to call getDiagnostics on both app.ts and dep.ts even though only dep.ts changed. However, if app.ts changes then we only need to call getDiagnostics on app.ts. This helps the best-case scenario but doesn't address the worst-case. If you change a file that nothing depends on it will be very fast. But if you change a file that everything depends on then it'll be the same performance as today.

A downside to this approach is that this list of files will be pushed through the loader again which can increase the "build modules" time, but we might be able to put in some optimizations for that.

Answer 16 · 2015-10-09T13:35:49.000Z

When you refer to using "watch", do you mean webpack --watch or webpack-dev-server? Do we need to use the in memory bundling of the webpack dev server to see the benefit?

Answer 17 · 2015-10-09T13:37:40.000Z

We are still seeing long load times even when using both --watch and webpack-dev-server. I think it is probably because all of our files are linked. One of our projects is a library, so all of the files in the project are linked in a hierarchy:

Components
|- Component1
|  |- Component1_1
|  |- Component1_2
|- Component2

etc, etc.
Each file imports its children and then exports them so that you can reference any component off of the components object. (via import { components } from 'components'; console.log(components.component1.component1_1);)

Answer 18 · 2015-10-09T13:38:24.000Z

From what you are saying, my understanding is that this forces ts-loader to recompile the whole project tree whenever one file changes because of the dependencies?

Answer 19 · 2015-10-09T14:16:05.000Z

@SonofNun15 , both webpack --watch and webpack-dev-server do the same thing in regards to compilation. The difference is only in output. The dev server uses an in-memory file system and serves the bundles via http along with some runtime code used for hotloading. The regular webpack just writes them to disk. Both of them stay in memory and watch files for changes. Performance should be nearly identical.

I am actually using a forked dev-server which writes files to disk and only use HTTP to get the stats (errors and warnings) for integrating them with the IDE. So it's basically the same thing.

Answer 20 · 2015-10-09T14:18:13.000Z

OK, I see. So webpack --watch keeps the modules loaded in memory, but I assume it does have to flush the whole file to disc on any change. This should be relatively quick given we are talking about KB - MB size files not GB. 😀

Answer 21 · 2015-10-09T14:24:34.000Z

Yes it is very quick. One other thing you should keep an eye on, and I don't know what you use in your project, isdevtool: 'source-map' which has to regenerate the source map for the entire bundle on the smallest change. That also increases build times linearly with project size, so ideally you should be using eval-source-map or something faster.

Answer 22 · 2015-10-09T14:56:22.000Z

@SonofNun15 When talking about --watch, you're still going to incur the initial build time but then subsequent builds should be a lot faster. I would be interested in hearing your numbers for the initial build and then for a change.

From what you are saying, my understanding is that this forces ts-loader to recompile the whole project tree whenever one file changes because of the dependencies?

Kind of, not exactly. ts-loader does get diagnostics from the entire program for every change, but that isn't the fault of how you've got your project structured.

Answer 23 · 2015-10-09T17:32:30.000Z

Current run time numbers are:
Initial load time: ~30 seconds
Reload time (webpack --watch): ~14 seconds

Bundle size is 1.85 MB

We were using dev-tool: "inline-source-map". I removed it before recording the above numbers.

Answer 24 · 2015-10-09T17:35:01.000Z

This is without minification, which reduces the bundle size to ~638KB

Answer 25 · 2015-10-13T09:19:56.000Z

@jbrantly , I opened this issue: microsoft/TypeScript#5192

Answer 26 · 2015-10-13T14:12:12.000Z

Interesting, thanks. Good to know they're aware of the problem of discovering deep dependencies. I still think there is a good possibility we can fix this now, but long term using an official API would clean things up.

I haven't had the time to really explore this yet due to other obligations but I definitely do want to fix.

Answer 27 · 2015-10-26T16:42:01.000Z

FWIW, I've been thinking on this a lot and I believe I know a way to improve the initial build time. I'm planning on trying it out sometime this week. I realize this issue is more about incremental build time but initial build time is important too.

I still have ideas on the incremental build time but they're not as fleshed out. I'm also planning on exploring that more.

Answer 28 · 2015-10-26T16:55:11.000Z

It's good to optimize both if possible, although the main bottleneck for the initial build time is probably not ts-loader, but webpack itself.

Answer 29 · 2015-10-26T17:44:38.000Z

Actually there is quite a bit of bottleneck in ts-loader right now. Using tsc to generate js and then just webpacking the js is significantly faster than using ts-loader for large projects. The issue (I believe) is that we are forcing TypeScript to re-synchronize everything pretty much for each file when we should only need to do it once in the common case. That's what I'm planning to address.

The idea for improving incremental build time is somewhat dependent on the resulting behavior of the above, so that's why I want to investigate that afterward. But the basic idea hasn't changed: we need to somehow track dependencies and then only get diagnostics for the changed files and their deep dependents. The trick is that "somehow" 😁

Answer 30 · 2015-11-05T13:28:38.000Z

As a potentially distracting aside, I recently switched from awesome-tsloader to ts-loader and for the same project, my build increased by a minute using ts-loader for an initial build.

Answer 31 · 2015-11-05T14:20:13.000Z

@raybooysen Not a distraction, it's good information to have. Once I implement the changes it'll be good to see how that affects your build

Answer 32 · 2015-11-05T14:22:33.000Z

Good news. Once the changes are in, I can switch between the two very easily (1 line change) and I can compare.

Answer 33 · 2015-11-06T16:34:29.000Z

I just opened #100 which improves initial build time performance. There's still work to do on initial build time (and obviously on incremental build time). Right now as near as I can tell a bunch of time is being taken up in getting diagnostics (still). I've noticed that tsc doesn't use the language service and instead just uses a program, even in watch mode. I'm curious about the performance characteristics of the two because tsc generally runs very fast even when reporting diagnostics. That's my next area of exploration.

Answer 34 · 2015-11-07T06:44:11.000Z

I just tried #100 on my project and the initial build time went from 70s to 6s. Thanks :)

Answer 35 · 2015-11-07T13:00:50.000Z

@phuvo Awesome! Thanks for the feedback. I just published those changes in v0.6.1

@raybooysen Could you try the latest release and see how it affects your project?

Answer 36 · 2015-11-07T13:30:09.000Z

22 seconds taken off the build. Awesome.

Previous build with ts-loader was 2m:03. Next build with 0.6.1 it is 1m:41

Answer 37 · 2015-11-07T19:32:22.000Z

@raybooysen Still a long time 😦 Hopefully I can cut it down a lot more. Do you happen to have a loader that executes before ts-loader?

Answer 38 · 2015-11-08T20:06:14.000Z

Yes we have a few loaders. I wish webpack would output timings for this.

Answer 39 · 2015-11-09T10:14:34.000Z

Actually saying that, I need to confirm if the loaders run before ts-loader. I think not. But this is a full build, I don't have a breakdown of how much time each loader is taking.

Answer 40 · 2015-12-11T15:38:25.000Z

Can we add an option to skip error checking at after-compile?
Since if someone uses incremental build, it is most likely it is used in development, and IDE already showing all those errors. So skip them in webpack output with the benefit of HUGE speed increase worth it, and does not looks hard to implement.
I've disabled after-compile hook in ts-loader and got an improvement from 6 sec to 15ms in incremental build.

Answer 41 · 2015-12-11T15:40:57.000Z

Pop up a PR. :)

Answer 42 · 2015-12-11T15:47:17.000Z

@Karabur , the IDE probably shows you errors in the file being currently edited, it won't show you errors in other files, caused by changes in the current file. In other words, it only helps with syntactic errors, not semantic errors. You will get those only by incremental build/ recompilation of the entire project. Because of this, I'm inclined to think that skipping error reporting provides little to no benefit. It may even prove detrimental if you discover errors later, because it will take you more time to figure out when and where you did something wrong.

Answer 43 · 2015-12-11T16:11:06.000Z

@raybooysen ok, will try to make a proper one.
@use-strict skipping error reporting provides no benefit, indeed. But getting compilation speedup like that has. And it is important when you have hot-reloading for example. And errors not hidden completely, you can set up your environment to easily see them at any time you would like to.

I am not sure about all IDE's but my (WS) shows errors for all files, not only for which I am currently edit. I dont need webpack output at all and have no benefit from loader forcing me to to wait 6-15 seconds on every single change to see app updated, but drawback is huge. Anyway it is matter of workflow, and since impact could be significant, it is always better an option to choose.

Answer 44 · 2015-12-11T16:33:13.000Z

IMHO, in the end, it still comes down to the performance of the incremental compilation. Anything larger than a few seconds is not acceptable, for me anyway. Skipping error reporting is not the right solution here. Getting the errors in an efficient way is.

I'm not sure in your case if those 6-15 seconds are just typescript compilation or overall time, including time spent by webpack. The usual bottleneck is dependency resolving, so there may be other things causing the large build time.

On a side-node, something to watch out for: WS has its own "language service" / compilation logic. It doesn't use the API exposed by typescript and they said it will never do so, for commercial reasons. Although rare, you might find yourself with compilation errors when the IDE says everything is fine and viceversa. WS10 at least was full of this kind of bugs and it was one of the reasons why I moved away from it.

Answer 45 · 2015-12-11T16:35:54.000Z

@Karabur Have you tried using the transpileOnly option?

Answer 46 · 2015-12-11T20:36:17.000Z

@use-strict of course it is workaround, not the solution. 6-14 is error processing step only. so, having all incremental time about 7 seconds I have like 1 second without. For relativeliy small project. Other people reported more time on that step.
Your info about WS is outdated and incorrect. You can use exact same TS compiler as ts-loader and have 100% of it's errors. But it is off-topic here, and other editors could have full type checking support as well (why they shouldn't?)
@jbrantly just tried it:
without any options: total build time: 10Sec (9 seconds overhead)
with transpileOnly: 3~4 sec (2-3 seconds overhead)
with errors disabled: 1 sec
transpileOnly seems to give good boost, and, since with transpileOnly there is also no errors printed, I think it is safe and correct to skip after-compile step completely if it is set to true

Answer 47 · 2015-12-15T10:11:26.000Z

I've checked code in more details and with transpileOnly options error gather is never called at all. Looks like things was messed up in my test runs, so that 1s time is caused by something else. I'll try to do more isolated and clean performance testing. So far transpileOnly looks like best option.

Answer 48 · 2015-12-15T13:54:52.000Z

Right. With transpileOnly the whole after-compile step is completely skipped. transpileOnly basically is the option to skip error checking (there is some minor error checking that goes on but I think it's only syntactic).

Answer 49 · 2015-12-15T13:59:34.000Z

I should note (probably in the documentation somewhere) that transpileOnly does have slightly different output behavior:

From here

imports\exports are not elided just because they cannot be resolved (since we have code for only one file)
const enums are treated as regular enums (even for locally defined const enums)
files are always emitted as modules even if they don't have top level export \ import \ require

Answer 50 · 2016-01-12T22:22:07.000Z

Any news or workarounds to make latest version of ts-loader as fast as 0.5.2?

Answer 51 · 2016-01-13T03:34:41.000Z

@panKt Not yet. I believe initial build time is actually faster now than in 0.5.2, but incremental build time still has the issue outlined here.

Answer 52 · 2016-01-13T03:56:07.000Z

@panKt, As a workaround you could use transpileOnly during development.

Answer 53 · 2016-01-20T09:40:27.000Z

I am having this issue too :( incremental compile gets slower and slower as the project gets bigger, I've resorted to using transpileOnly for now but it's bad because I keep having to remember to toggle it to check for errors :(

Answer 54 · 2016-01-20T11:30:11.000Z

Is it getting slower in relation to how much code you're adding? This
would be expected then.

On 20 January 2016 at 09:40, Wael notifications@github.com wrote:

I am having this issue too :( incremental compile gets slower and slower
as the project gets bigger, I've resorted to using transpileOnly for now
but it's bad because I keep having to remember to toggle it to check for
errors :(

—
Reply to this email directly or view it on GitHub
#78 (comment)
.

Answer 55 · 2016-02-02T15:41:46.000Z

Had the same issue, setting "sourceMap": false in tsconfig.json helped tremendously (much more than expected).

Answer 56 · 2016-03-20T23:11:36.000Z

Setting "sourceMap": false doesn't change much in my case. But setting transpileOnly brings down the compile time from 2.5s to 700ms. This might now seem much, but it is actually 3 times faster.

Answer 57 · 2016-03-27T00:31:03.000Z

awesome-typescript-loader stores an internal dependency graph. Would this help at all?

FWIW, both ts-loader and awesome-typescript-loader seem to have an issue with optimize-assets taking a long time :(