huge memory usage & crash large project
endiliey opened this issue · 0 comments
Motivation
Found this upon investigating facebook/docusaurus#1782 and profiling memory usage in Chrome dev tools
In a large project (~> 400 pages), this plugin will consume huge memory and crash it due to the fact that webpack 4 outputs a large amount of data with its stats.toJson() by default
The error will be
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
Even increasing to 8gb heap memory doesn't work.
Upon removing this plugin, the website can build flawlessly (even without increasing memory limit)
Recommendation
Possible improvement
- I believe we dont need full output of stats.toJson.because it can be very resource-heavy. We should minimize the amount of data retrieved from the stats object by passing needed stats https://webpack.js.org/configuration/stats/ instead
Refer to stats.toJson source code.
https://github.com/webpack/webpack/blob/1a773336187ec0218e060d453f801bc73ab98a6e/lib/Stats.js#L104
Maybe something like
this.stats = compilation.getStats().toJson({
all: false,
chunks: true,
entrypoints: true,
chunkModules: true,
reasons: true,
}, true);
- Rework the logic of getting chunk origin
react-loadable-ssr-addon/source/ReactLoadableSSRAddon.js
Lines 124 to 151 in 1b0c0a0
reasons
stats is very heavy and consume a lot of memory (still crash the site)
Can we use a more memory-friendly stats option and achieve same result ?