The PreserveModuleNamePlugin doesn't play well with DllReferencePlugin

Question

The PreserveModuleNamePlugin doesn't play well with DllReferencePlugin

jack4it opened this issue 8 years ago · 26 comments

I managed to put all core dependencies into a DLL bundle. But when generating the app bundle where the DllReferencePlugin is used, the logic in PreserveModuleNamePlugin runs into multiple issues, mainly due to the referenced modules are DelegatedModule instead of NormalModule, and miss the resource and rawRequest properties that the PreserveModuleNamePlugin expects. Any thoughts on how to fix this?

jack4it commented 8 years ago

Awesome!

Answer 1 · 2017-02-17T09:49:36.000Z

Coincidentally this was reported to me yesterday by another user. And I had a long brainstorm about it.

Right now, DLL bundle don't really work.
Well, static imports work.
There's a bug but I could easily make Aurelia imports from your app -> dll bundle work.

But the hard part is that internal Aurelia dependencies (dll -> dll) won't work and this requires complex changes to aurelia-loader-webpack, it's not just a matter of bundling.

I'll have a go at it but it's a tough nut to crack so it'll take time.

Let me know if fixing Aurelia deps from app -> bundle would be enough for you. That I could fix quickly.

Answer 2 · 2017-02-17T11:05:24.000Z

I worked around the error in the PreserveModuleNamePlugin via a debugger, got the bundling itself work. However, the browser then reported a module from the dll is not found. This is from the loader logic. Are these all about the app->dll imports you mentioned?

Do u have a sample scenario of dll->dll? Would love to understand more and potentially help in the future

Answer 3 · 2017-02-17T22:01:14.000Z

The hard problem here is that DLL have their own webpack context. They basically create a global variable that is it's own __webpack_require__. aurelia-loader is only aware of the main one (your app bundle).

Consider a library aurelia-grid. It contains index.js, which does .globalResources on grid.js.
Assuming the library makes use of PLATFORM.moduleName, you'll end up with 2 modules inside a DLL: index and grid. All fine so far.

Now consume this DLL from your app. In app.js you'll probably do .plugin("aurelia-grid/index"). Let's assume a fixed PreserveModuleNamePlugin so that bundling works.
In your main bundle, you'll end up with two modules: app and aurelia-grid/index, which is just a proxy that returns aurelia_grid("index").

The big issue now is that there are two "namespaces" at runtime, your app bundle, which aurelia-loader is aware of, and aurelia-grid.

When the app start, Aurelia will request aurelia-grid/index. Because this was a app -> grid dependency, it's in the app bundle and will work.

During execution of aurelia-grid/index, Aurelia will try to load aurelia-grid/grid. And because this is an internal grid -> grid dependency, it exists in the aurelia-grid namespace but not in the app namespace. Because everything aurelia-loader does happens in the main bundle, this load will fail.

Answer 4 · 2017-02-18T02:22:23.000Z

For what is worth, app -> dll dependencies should be fixed.
So loading stuff from DLL will work, but you will still get failures at runtime if the contents of DLL have intra-DLL dynamic dependencies.

For example:

using aurelia-pal-browser from a DLL should be ok.
using aurelia-templating-resources won't work.

That's some progress.
As explained above the second half is harder and needs aurelia-loader-webpack changes. I'll try to tackle that but it'll take more time.

Answer 5 · 2017-02-18T02:45:18.000Z

Cool, where can I see the fix and/or get the latest bits and try them out? Thanks for the quick turnaround.

Answer 6 · 2017-02-18T12:33:58.000Z

The aurelia-webpack-plugin#v2.0 branch was updated.

Answer 7 · 2017-02-18T19:18:46.000Z

I did some research today. It's even worse than I thought.

When aurelia-loader is inside a DLL (which is what people are typically trying to do with a vendor.js DLL), it only has access to the __webpack_require__ that lives inside the DLL. Webpack spills absolutely nothing in the global scope.

So, even though your app is in the main bundle and could in theory be hacked to have access to everything, the actual loading happens through aurelia-loader-webpack which lives inside vendor.js and it has no access at all to your app code. This of course means nothing works.

The current status is:

Consuming non-Aurelia libs and code through DllReferencePlugin works.
Moving Aurelia code into a DLL is not supported.

I think we could make it work as long as aurelia-loader is kept inside the main bundle. Of course, this implies that anything that depends on aurelia-loader (and a lot of Aurelia core does) can't be in a DLL either. This limitation reduces the usability of the feature quite a lot, to the point I'm not sure it's worth doing.

Another option would be extreme hackery and spill stuff in the global scope at build time, modify aurelia-loader-webpack to grab this global info when it is created and use this to bridge worlds. It won't be pretty for sure!

@niieani, @EisenbergEffect thoughts?

Answer 8 · 2017-02-18T20:23:55.000Z

What is a DLL bundle? Are we talking about a .NET or Native Windows dynamic, link library here?

Answer 9 · 2017-02-18T23:56:50.000Z

@EisenbergEffect No, it also refers to a webpack library, i.e. some code that has been built separately.
It's not just about multiple files, but distinct compilation units.

Some people use webpack's DllPlugin to create a bundle of all their static libs (including Aurelia) in one compilation pass.

They then only compile their own code, bringing in the pre-compiled libs with DllReferencePlugin.

I might have a new crazy idea to make this work... Have to do some research and trials.

Answer 10 · 2017-02-19T09:26:14.000Z

With this hack PLATFORM.moduleName("aurelia-templating-resources/compose"); for the list of core dependencies specified in my app entry main.ts file. I have made this work. I can't believe this :) But do let me know if you want to see my full config(s) for potential reference. Thanks for researching on this and any crazy ideas trying to make this happen. Dll/DllReference is sort of a must have for many teams trying to separate the vendor and app bundles. So this is really useful

Answer 11 · 2017-02-21T01:16:14.000Z

Good news, I have a proof of concept that works. I should be able to pull this off.

Answer 12 · 2017-02-26T20:25:06.000Z

OK I finally managed to get everything working properly. Don't ask, just assume it's ✨ magic ✨ 😣

@jack4it I would appreciate if you could try it.
There's some information here, although there's not much required:
https://github.com/jods4/aurelia-webpack-build/wiki/Using-Webpack-DLL
⚠️ Until the next release of aurelia-bootstrapper there is one manual step though, please look at that page.

I have made a demo project demos/05_DllReferencePlugin as well, which works.

Answer 13 · 2017-02-27T20:33:18.000Z

Hey @jods4, here is an error I just got after a fresh npm install and changing the configs as suggested in your wiki page. Let me know if you want to see my config details. I can pass that over an email or gitter 1:1

C:\__prj\samm-webpack\src\Www\node_modules\aurelia-webpack-plugin\dist\PreserveModuleNamePlugin.js:35
                    module.meta["aurelia-id"] = id = id.replace(/\\/g, "/");
                                              ^

TypeError: Cannot set property 'aurelia-id' of null
    at Compilation.compilation.plugin.modules (C:\__prj\samm-webpack\src\Www\node_modules\aurelia-webpack-plugin\dist\PreserveModuleNamePlugin.js:35:47)
    at Compilation.applyPlugins1 (C:\__prj\samm-webpack\src\Www\node_modules\tapable\lib\Tapable.js:75:14)
    at sealPart2 (C:\__prj\samm-webpack\src\Www\node_modules\webpack\lib\Compilation.js:582:9)
    at Compilation.applyPluginsAsyncSeries (C:\__prj\samm-webpack\src\Www\node_modules\tapable\lib\Tapable.js:131:46)
    at Compilation.seal (C:\__prj\samm-webpack\src\Www\node_modules\webpack\lib\Compilation.js:570:8)
    at C:\__prj\samm-webpack\src\Www\node_modules\webpack\lib\Compiler.js:474:16
    at C:\__prj\samm-webpack\src\Www\node_modules\tapable\lib\Tapable.js:225:11
    at _addModuleChain (C:\__prj\samm-webpack\src\Www\node_modules\webpack\lib\Compilation.js:472:11)
    at processModuleDependencies.err (C:\__prj\samm-webpack\src\Www\node_modules\webpack\lib\Compilation.js:443:13)
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
    at process._tickCallback (internal/process/next_tick.js:98:9)

Answer 14 · 2017-02-27T21:01:40.000Z

just to be sure if this is something weird or something "expectable", can you tell me what the module is?

Answer 15 · 2017-02-27T22:01:42.000Z

I did some webpack source code inspection. Seems to me that the only way meta could be null is if your module is in error (check module.error and module.errors). Doesn't mean crashing is a good thing to do but would explain why.

If you can confirm I should be able to fix that easily.

Answer 16 · 2017-02-27T22:16:20.000Z

Also getting the same error as jack4it, i am trying to load aurelia-dialog.

I had to add the following to my webpack config file, not sure if anything else is required?

new ModuleDependenciesPlugin({
"aurelia-dialog": [ "./ai-dialog", "ai-dialog-header"]
}),

Answer 17 · 2017-02-27T22:21:51.000Z

@phinett not sure either but if you fully test out aurelia-dialog let me know please. It's one of the few "core" aurelia libs that have not been approved with the new webpack plugins yet.

Back to the initial issue: any idea why module.meta was null?

was the module in error (check module.error and module.errors)?
otherwise what kind of module was it?

Did adding the aurelia-dialog dependencies fix your issue? If yes I'm really unsure why.

Answer 18 · 2017-02-27T22:35:23.000Z

Sorry no adding the dependencies didn't fix the issue that was something else i was having trouble with, most probably irrelevant to this issue actually so ignore that.

With aurelia-dialog i am now getting this error when compiling with webpack:
module.meta["aurelia-id"] = id = id.replace(/\/g, "/");
Cannot set property aurelia-id of null

I am not sure where to check module.error / module.errors ?

Answer 19 · 2017-02-27T22:40:49.000Z

you'd need to debug your build for that or possibly add a console.dir right here.

Answer 20 · 2017-02-27T23:16:05.000Z

Here is a screenshot of the error. It's breaking on a module with id "tslib/tslib.es6.html". I'm using this TypeScript library to reduce the TS transpilation size. In my other repo where I have a workaround to make DLL/Reference work, without this fresh npm install, this error/warning also presents but doesn't break PreserveModuleNamePlugin. The warning there is something like this:

ERROR in ./~/tslib/tslib.es6.html
Module parse failed: C:\__prj\Sammamish\src\Www\node_modules\aurelia-webpack-plugin\dist\html-requires-loader.js!C:\__prj\Sammamish\src\Www\node_modules\tslib\tslib.es6.html Unexpected token (1:0)
You may need an appropriate loader to handle this file type.
| <script src="tslib.es6.js"></script>
 @ ./~/tslib/tslib.es6.js
 @ dll vendor

Answer 21 · 2017-02-27T23:27:08.000Z

ok I get it.
There's an error no this module so this is why meta is null. I'm going to fix it.

There's another issue here that I'm very unhappy about but I need some time to reflect upon it.

Answer 22 · 2017-02-27T23:30:37.000Z

I pushed the fix for the compiler crash.

Answer 23 · 2017-02-27T23:49:25.000Z

@jack4it and a workaround for your tslib problem is now pushed on the v2.0 branch as well.

Answer 24 · 2017-02-28T01:50:07.000Z

All right, cool, now the bundling works. Awesome. One minor thing is that, the DLLRef plugin options allow manifest to be just a path string instead of having to be a JSON object. On line 113 of AureliaPlugin.js, it will throw in the path string case. Other than that, everything works well at this point. Thanks!

Answer 25 · 2017-02-28T10:16:47.000Z

@jack4it I'll have a look.