Make bower Github Release Asset aware

Question

Make bower Github Release Asset aware

trek opened this issue 10 years ago · 40 comments

Apologies if this idea has been floated before. I did a cursory search and couldn't find anything.

The impetus behind this proposal is a long exploration of the painful JavaScript modularity story. I'll save you the read and summarize the most salient points:

In Bower the average JavaScript library dependency depth rounds to 0.
Most JavaScript is not authored in the format that people consume it.
Modularity only Just Works™ when your each item in your dependency tree has a standard way to expose itself.

These general problems also apply to other asset types (e.g. css, images), but I'll focus on JavaScript for this proposal.

Dependency Depth

@rayshan was kind enough to give me a data dump for the last 30 days of bower installs. I'm still playing around with that data, but I think I have enough to make some points. Typical dependency depth for libraries published to Bower is 0. I don't have any data on npm, but just looking into some node based projects on my computer, it's not uncommon to see dependency depths of 5 or more.

One exception to this in Bower is the clusters of libraries around foundational projects like jQuery, Ember, or Angular. Here, dependency depth is between two and three. This appears to be because these projects' communities either have a shared build ecosystem and/or a mandated modular format (ES6 for Ember, the $ namespace for jQuery, angular.module for Angular, etc).

JavaScript outside these silos tends to lack (or not expose) modularity. This lack of modularity is leading to fragmentary solutions which complicates the process of getting dependencies into projects without a lot of additional effort on the part of the package consumer.

The Gold Standard for this should be systems like Rubygems and npm where coarse (package-level) and fine-grained (specific object import) dependency is so frictionless, you'd be crazy not to use it.

To summarize:

npm install foo

Gets the foo package as a copy locally

var aFoo = require('foo');

Is really a shorthand for the loader to go find node_modules/foo/package.json, look for its main entry, and bring the contents of that file into the execution environment.

In Bower the process is more like:

bower install foo

# lots of complicated custom tooling to convert
# between module formats or wrap un-modular
# code in IIFEs for safety or wrap in module shims

require(["foo"], function(foo) {});

Problems

The lack of that common module system is forcing authors to engage in various painful-to-maintain patterns:

Checking in compiled code (or multiple versions of it) into source control. A.k.a. the dist/ pattern. Pain points: having a main entry in a package manifest is basically meaningless, forgetting to build before publish leads to problems where a tagged version's dist/ is still the old code, not every project uses the same folder for builds, so each consumer has to go find where code is.
Creating special module-specific forked repos that are not associated with the main project (https://github.com/amdjs/backbone, https://github.com/amdjs/underscore) and then publishing those as new Bower packages. Problems: "This branch is 16 commits ahead, 489 commits behind jashkenas:master".
Creating official shim repos https://github.com/components/ember. Problems: polluting Github with "read-only" projects, maintenance pains around people not knowing that the repo isn't the real one and opening issues and pull requests against it.
The "¯\_(ツ)_/¯" approach of shipping one module system. I chatted with @jeremyckahn about how he handles the module problem for https://github.com/jeremyckahn/rekapi/ and he basically said "what module problem?". Turns out he ships only in AMD/one global variable exposed and other format users are just not in scope. This is also true for everyone who publishes as CJS and says "just use browserify!".
Embedding dependencies: jQuery embeds Sizzle. Ember embeds RSVP.js, router.js, route-recognizer.js, etc (about a dozen libraries). This makes combining these libraries complicated. If I build a library on top of Sizzle but not jQuery and you also pull jQuery into your project you effectively have two copies of Sizzle. You can work around this by creating custom builds (jQuery compat and non-jQuery compat) but that just adds to the "where do builds go!?!" problem.
Thinking UMD will help. (It does if your dependencies have no dependencies – "UMD works great when you have no dependencies in your lib" – which contributes to the overall shallow dependency tree).

Ideal Solution

We all author in ES6 modules and compile to AMD for runtime. When node 0.12 ships it lets people use ES6 modules natively. The node community rapidly shifts over. By December 2015 all major browsers parse ES6 modules natively. I get a pony.

Actual Solution.

It's been 5 long years already. We're not going to convince everyone using one module system in the near future. Maybe in 2016 or 2017. Nobody likes alternate module systems. Everyone thinks their module system has "won" or will "win". Everyone is wrong.

Until that all gets resolved, I'd like to suggest we make Bower aware of Github Releases and, more specifically, release assets. Release assets:

You can also attach binary assets (such as compiled executables, minified scripts, documentation) to a release. Once published, the release details and assets are available to anyone that can view the repository.

The 10,000ft view of how this could work:

Authors can remove built code from source control.
Authors can author in whatever module format or language they prefer.
Authors can publish builds as release assets that transpile from their preferred authoring format into executable JavaScript wrapped in multiple module syntaxes
Authors can use what ever build tools they prefer to prepare these publications
Consumers can specify preferred module formats in their projects. Bower will attempt to provide this to them.
Consumers will get the default behavior (plain source checkout) if an asset of the preferred format is unavailable.

The lower level view of how consumers use this (which we can bikeshed):

In a project .bowerrc, I can specify acceptable build types in order

  {
    "buildAssets": ["es6", "amd"]
  }

or maybe grouped by asset type?

  {
    "buildAssets": {
      "javascript": ["amd", "global"],
      "css": ["sass", "css"],
      "icon": ["svg"]
    }
  }

In addition to sha/tag mappings, the bower local cache gets populated with
available build assets by name for each release when using the GitHubResolver.
When a consumer attempts an install Bower checks builds by name as part of the resolution phase
If a build is available for the resolved release version, that build is downloaded instead of the release build.
So for https://github.com/trek/multi-publish-example, asking for amd gets you
https://github.com/trek/multi-publish-example/releases/download/v0.0.1/amd.zip instead of https://github.com/trek/multi-publish-example/archive/v0.0.1.tar.gz
If no matching build is available, the current behavior of just obtaining the source code applies.
Builds can be specified explicitly and stored in the bower file so this check can be skipped later.
```
bower install jquery@amd
# or
bower install jquery#2.1@amd
```

Some problems with this proposal:

You need to use the Github API, which is rate limited to 5000 requests/hr. With caching this may not be an actual problem in the wild.
Projects that ship multiple asset types (e.g. Bootstrap). Do they need to make a build for every asset type combination? Convert into a metaproject that references one JavaScript, one CSS, and one Font project? Maybe they just don't participate at all?
What is special about module formats over, say, precompiled-code formats (coffee, clojurescript, etc)? I'd argue that because they represent the standard way a library exposes its public interface, module systems have more in common with wrapping code in IIFE's than compiling from coffeescript to executable JavaScript. In my mind, if you authored in coffeescript, that code is black-boxed inside the module so your publish phase would compile from coffee to javascript, then convert your authoring module format to all compatible module formats. Other people (@domenic) disagree.

Answer 1 · 2014-11-20T17:27:07.000Z

cc @timrwood who is experimenting with ES6 module authoring for moment.js https://github.com/timrwood/moment-es6

Answer 2 · 2014-11-20T17:49:51.000Z

👍 to the careful thought, background, and reasoning. It's long been clear that anyone arguing a single module (or packaging) format is going to emerge as dominant—especially across all of the contexts in which JavaScript is written—is probably delusional or trying to sell you something.

I think Github releases and its API are one potential path (I'm woefully unfamiliar with what they have exposed via the API). I have to ask:

is there an opportunity for a decentralized or at least re-implementable alternative to Github Releases to accomplish this? Realizing Github is incredibly dominant, especially insofar as bower is concerned, I'd be content just to know someone could write an adapter to another release source of record in a pinch.
is this an opportunity to specify more concrete metadata about assets? Getting the types of assets and how they reveal themselves is certainly useful, but the concept of an entry point is difficult to pin down with a lot of projects that currently publish full source, minified source, localized sources, plugins, CSS, fonts, images, and so forth. Until Bower or another tool finds a way to popularize people formalizing this metadata, I worry improvements like this won't eliminate the need for custom build configurations for each one-off dependency.

Answer 3 · 2014-11-20T18:08:10.000Z

I like the proposal. I am currently working on a project which has put off listing on bower explicitly because we did not want to check in compiled or concatenated assets. If bower were aware of GitHub releases this would alleviate a huge pain point for us.

Answer 4 · 2014-11-20T20:34:00.000Z

@searls

is there an opportunity for a decentralized or at least re-implementable alternative to Github Releases to accomplish this? Realizing Github is incredibly dominant, especially insofar as bower is concerned, I'd be content just to know someone could write an adapter to another release source of record in a pinch.

For sure. Each resolver would need to figure out how to get this meta data from a package source, but nothing is stopping a package source from exposing this.

is this an opportunity to specify more concrete metadata about assets?

I wish, but I don't think the bike shedding on that would even stop. My hope is that, for JavaScript at least, we can get main to mean "have your loader/resolver start at this file and then walk the dependency tree from there".

Minification is an interesting bit. I feel like people who prefer global builds also prefer to have a minified option available to them.

People who care about module systems are more likely to use a build tool and know that minification is not enough. They're probably most interested in tree-shaking and whole-app minification rather than having dependancies pre-minimized for you. Minification is a fickle process, but I've noticed that delaying minification as late in the build process as possible often leads to smaller builds as the minifier gets more aggressive finding repeated strings in your apps and across all dependencies.

Answer 5 · 2014-11-20T21:47:36.000Z

Uh.. I agree about the problems, but not about the solution. You said "We're not going to convince everyone using one module system in the near future." but at the same time you want authors to create builds for different module systems. Authors don't care. Only consumers care. We can't use GitHub builds for that, because only authors can create them.

Btw. did you read #1520?

Please think about other solution, so builds can be automated, and consumers can define them.

Answer 6 · 2014-11-20T22:35:40.000Z

Authors don't care. Only consumers care.

I'm a library author and I care. This problem literally keeps me up at night. If I didn't care I'd just say "I write in CJS and publish on npm. Just go user browserify. It works." I know many, many other authors who care.

Please think about other solution, so builds can be automated, and consumers can define them.

Putting this onus on consumers doesn't make a ton of sense to me. When a compile fails and the consumer sees an error, they're the least likely person to know how to fix my library.

Answer 7 · 2014-11-20T22:48:20.000Z

I believe you care, but ultimately many (most?) of authors don't. Even if they do, they don't have the time to do so. Even you: https://github.com/trek/grunt-neuter/issues/48 I hope you get what I mean, and I don't mean to offend you.

As a consumer I don't want to wait month for author to fix their bower.json or create build for my module system. People write custom solutions for those problems, like bower-installer for overriding bower.json entries or SystemJS for loading modules in any format (AMD, CJS, ES6).

Answer 8 · 2014-11-20T23:07:47.000Z

I'm a library author and I care.

Answer 9 · 2014-11-20T23:08:23.000Z

Even if they do, they don't have the time to do so.

That's fine. For authors who don't care, the current behavior remains unchanged.

Even you: trek/grunt-neuter#48 I hope you get what I mean, and I don't mean to offend you.

I fail to see how this is related to the issue at hand. Maybe you can help me understand why you feel its related to publishing built assets? He's asking for an update to a Changelog to reflect the 0.6 release (there weren't really any changes of note). The commit he's hoping will go out actually introduced a bug (one that never got resolved), which is why it is not yet released and a 0.7 has not been issued. To me this all sounds like rational and responsible authorship.

As a consumer I don't want to wait month for author to fix their bower.json or create build for my module system.

That's fine. If an author doesn't participate the current behavior still occurs.

Answer 10 · 2014-11-20T23:10:21.000Z

For what is worth, I also care about this as both a library author that wants to provide a good user experience, as well as a library author that has to consume other libraries as dependencies.

Answer 11 · 2014-11-20T23:14:12.000Z

Hi Trek! Lemme say thanks for digging hard into this and spending plenty of brain-time with it.

The a-ha moments of your Medium article & this proposal is discussing that consumers just want it to work. They don't want to deal with build processes or compilation rigamarole.

I share sheerun's concern about authors. They don't want to do extra work either.

What I was hoping to see was a proposal that puts all the responsibility on the package manager. Something like...

Authors code how they want
They identify their format in manifest.json, and list what hooks they expose
Consumers request the format they want in their consumption manifest.json
The package manager does the heavy lifting between the two formats

This is what I'd like to shoot for. Reading the proposal, the part I get hung up is how the author has to specify the build assets. Is this something the package manager can facilitate? Maybe I'm being idealistic.

@sheerun @trek I think we got into a straw man fallacy rat hole there. Authors care. Consumers care. Let's talk about package management.

Answer 12 · 2014-11-20T23:28:19.000Z

I believe @trek addresses the issue of where the transformation logic goes here.

If cli-tool can’t transform from one format to the other (e.g. circular dependencies that ES6 can handle, but CJS cannot or expressions in require statements that make resolution non-deterministic) warn the author and allow them to either fix or choose to skip publishing to a particular consumption format.

If there is an error when moving from one format to another, letting the author know that so they can fix it before they even publish sounds like the best approach.

Answer 13 · 2014-11-20T23:33:59.000Z

@desandro

They identify their format in manifest.json, and list what hooks they expose

That sounds basically the same as bower/spec#10 , which never got fully resolved due to Bower's in(activity|decisiveness).

Answer 14 · 2014-11-20T23:48:22.000Z

I agree with most of the pain points surrounding Bower that @trek mentioned, and feel that's one of the reason you see a lot more granularity in npm modules (single function modules, no fear of dependencies, etc).

I also agree that there is no clear "winner" in terms of build tool or module pattern, and they all have their own uses and audiences.

But if I had to publish each of my modules to every different format, I probably wouldn't get any work done. ¯_(ツ)/¯

Might not be a big deal for somebody maintaining a dozen libraries that span a broad range of functionality, but for the "Unixy" types who have hundreds of modules, it would be hell on Earth. This would have to be automated down to a tee (as in: I don't even notice the process when creating and publishing modules/patches).

@desandro I like the idea of "code how you want" and using a manifest. Here's my main concerns with that:

nobody seems to agree on manifest files. Is it package.json? Or bower.json? Or fad-tool-of-the-week.json?
nothing is really black and white. e.g. bower init asks whether your module types are "es6" or "node" -- what if you're writing a CJS node and browser module that uses an ES6 source transform?
Bower is a Wild West in terms of manifest inconsistencies. Even major libraries like ThreeJS and PixiJS were not following common guidelines up until recently. I don't think this will change unless the manifest config is more strictly enforced.

Answer 15 · 2014-11-20T23:50:22.000Z

@desandro

Maybe I'm being idealistic.

To be honest, after a "interesting? but not yet" from npm, I backed off of my original idea a bit before pitching it here and scaled it down to just enable authors[1] to start working. It's a start, but not the end.

I'd love for the package manager and its cli tool to make the process 100% opaque to both author and consumer. That's a tall order for an MVP. And, going off the Bower README, seemed antithetical to Bower's principles (emphasis mine):

Bower offers a generic, unopinionated solution to the problem of front-end package management,
Bower runs over Git, and is package-agnostic. A packaged component can be made up of any type of asset, and use any type of transport (e.g., AMD, CommonJS, etc.)

There's much less structural standard on Bower compared to the typical npm package. I'm not sure that Bower can detect all the various ways people have organized their code which is why I abandoned that idea specifically for Bower and left format publication up to the author.

Everyone wants this to stop being a problem but I'm basically fighting a circular war right now. Authors I've approached (except @timrwood from moment.js, @mholt from papaparse and folks on Ember.js like @rwjblue) basically say "I don't want to transpile because there's no way great to consume multiple formats right now". People running package registries tell me "there's no reason to support multi-output builds because most authors don't do multi-output builds."

Me, I just sit here being all

I'm convinced from conversations with authors, consumers, and from discussions on issues like bower/spec#10 that one side of this has to happen before the other. The fact that Github has the infrastructure in place to enable it for authors[1] made me pick this as the first stab.

I don't think Bower can mandate package format at this point. Possibly not ever. And that's OK! If we can get Bower to enable consumption of builds, we can a) build consumption conventions on top of that and b) write a cli tool that makes mulit-output publish easy for authors and enforces some kind of structural pattern. I think my participation on Ember core is enough proof that I have no problem telling people "use this convention. Stop bike shedding"

Maybe one day that tool can get merged into the bower version command, but it seems like it's better to iterate on that tool outside of Bower first.

But, hey, if you want that as part of Bower, full efing steam ahead!

[1] the ones who care.

Answer 16 · 2014-11-21T00:12:48.000Z

@mattdesl do the libraries you author have dependencies? If not and you're willing to

author in ES6 modules
publish to github
use release assets

I will work to automate this for you. I even publish to npm in cjs.

Answer 17 · 2014-11-21T00:59:16.000Z

do the libraries you author have dependencies

Yeah, most of them have dependencies.

If release assets could be integrated into my existing workflow/tools so that it was painless to do on every patch (which includes countless readme updates) then I would probably consider it.

My current reasons for not authoring in ES6:

module interop is concerning
I would need to transpile on prepublish to support Node. Not the end of the world, just additional boilerplate/setup
module entry point would diverge for Node & Browser. I like that most of my modules have a single entry point right now.
ES6 introduces a bundler-specific source transform. Most of my modules don't depend on any browserify transforms, which makes them easy to consume by tools like Webpack/Duo/etc
dependencies ebb and flow during development; sometimes they will be added or removed late in the game as I find commonalities with new modules. I can rarely say "this module will never need dependencies"

Answer 18 · 2014-11-21T01:20:15.000Z

Yeah, most of them have dependencies.

@mattdesl <3. You might have to live with this problem a bit longer until a) those authors agree to multi-publish b) we're all living in future where all javascript has a a single module system.

Answer 19 · 2014-11-21T01:43:56.000Z

@trek I've read your article. It's really nice writeup, kudos.

I really agree on following points:

Builds in source code are awful
As an author I want to use only one module format to develop in
As a consumer I want to choose module format of all of my dependencies

I cannot completely agree that builds need to be hosted. As long as client can perform build by itself, we're good. Optional hosted builds like in Homebrew (bottles) or Nix are very convenient.

@mattdesl Agrees that creating such hosted builds is very tedious task. It needs to be automated as much as possible. Builds can be performed by external service or by clients. In either case we need clear definition how to build given component to selected module system. Client builds are especially important when someone chooses to use source of component instead of its release.

I believe it's in consumer's interest that his dependencies are in the same module format. Author shoudn't even care there are 10 different module systems out there, and arrange builds for them. Specifically I believe that it's consumer's job to define build tasks for thor their dependencies for their module system (e.g. wrappers for components that export only globals).

Because whole tree of my dependencies need to be in same module format, we need a way to allow only components that can be built to given module system. Ecosystems in npm solve this issue pretty nicely. Maybe bower could do the same (amd registry, cjs registry, globals registry, less registry, sass registry). If you declare you use cjs and less, you can use only dependencies from those registries.

All to all I am pretty sure if solution exists, it consists of:

Build definitions created by consumers (the same as in Homebrew)
Automated optional hosted builds, not managed by authors (the same as in Homebrew)
Separate registry for every module system (ecosystems)

Answer 20 · 2014-11-21T01:55:08.000Z

You might have to live with this problem a bit longer until a) those authors agree to multi-publish b) we're all living in future where all javascript has a a single module system.

That's the beauty of npm -- it encourages a very high level of reusability, even if your dependencies are just tiny functions ([1][2][3][4] etc). Many of the existing attempts to cross module boundaries gives me anxiety about these deep dependencies. 😢

I don't think we'll achieve "singularity" until Node supports ES6 by default, and ES6 plays nicely with CJS for deep dependencies (and vice versa). By the time that happens (if it ever does), npm will be so saturated with stable tools and modules that it would be asinine to suggest refactoring them to support some newfangled tooling.

Answer 21 · 2014-11-21T02:23:50.000Z

All to all I am pretty sure if solution exists, it consists of:
Build definitions created by consumers (the same as in Homebrew)
Automated optional hosted builds, not managed by authors (the same as in Homebrew)
Separate registry for every module system (ecosystems)

Sounds like a good idea too. Is there place where we can track this project?

In effect doesn't the creator of a build definition just become an author, with all the same problems of authorship? See https://github.com/amdjs/backbone, https://github.com/amdjs/underscore, etc which are woefully out of date.

https://github.com/lodash/lodash-amd and https://github.com/components/ember are up to date, but that's because they're maintained by the original authors. To me this is solid evidence that authors do care, are willing to publish to multiple formats, and would be willing to use an official solution instead of publishing and maintaining extra shim repos.

Regardless, the presence of one good idea doesn't invalidate others, so I don't see how a Homebrew style solution really has any bearing on the proposal. By all means, go give that a try too!

Answer 22 · 2014-11-21T02:26:01.000Z

That's why it's important builds are automated, and consumers/authors create only build definitions.

Answer 23 · 2014-11-21T02:33:22.000Z

That's why it's important builds are automated, and consumers/authors create only build definitions.

👍 can't wait to see this project. It also seems like a good idea. I'm not particular about how this problem gets solved, just tired of it being a problem for the last five years.

Answer 24 · 2014-11-21T02:52:39.000Z

I think first two things to agree on are build system and structure of GitHub repository for definitions.

Probably gulp, it can quickly compile whole dependency tree in-memory
Maybe something like:

.
├── angular
│   └── amd
│       ├── 1.1.0.js
│       └── 1.1.1.js
├── bootstrap
│   ├── less
│   │   └── 1.1.0.js
│   └── scss
│       └── 1.1.0.js
└── react
    ├── amd
    │   ├── 1.0.1.js
    │   └── 1.0.2.js
    └── node
        ├── 1.0.1.js
        └── 1.0.2.js

The contents could be just gulp plugins for building given asset type of given component version.

Another option is to skip second level and embed types in build files (maybe even better).

Answer 25 · 2014-11-21T12:20:43.000Z

@sheerun I like your idea, but it seems like it belongs on its own repo. I'm not sure how it relates to people consume release assets via Bower.

Release assets don't mandate any particular task library or build system. Part of my goal for this proposal was to let authors begin using this system with whatever current build process they have.

People are opinionated about their preferred build tool as they are about their preferred authoring formats. Not everyone can or will move from their current build process to a single "standard" one. I wish that weren't the case, but I have to work in the world that we have not the one I wish we had.

Answer 26 · 2014-11-21T13:27:43.000Z

OK, so let's talk how bower can enable such system to be built. But please let's not focus on GitHub release assets. Let's focus build definitions and release builds in general, not necessarily hosted on Github and managed by authors. Maybe it can be built first as bower plugin.

Answer 27 · 2014-11-21T14:02:00.000Z

@sheerun I'm interested where you go with that idea, but not interested in playing a part. I'm nicely trying to hint "please stop hijacking this thread with a totally different idea".

If you're a "No" on this idea of release assets, I think that's understood. If you have questions or comments about the technical specifics of using release assets, I'm happy to discuss.

Answer 28 · 2014-11-21T14:12:31.000Z

If people are worried "nobody will do this unless it's automated", the process trivially easy to automate. The tools exist and have existed for a while. He's an example of transpiling from ES6 to AMD, CJS, Globals and publishing to npm: https://github.com/trek/multi-publish-example

The compile code is 20 lines https://github.com/trek/multi-publish-example/blob/master/Brocfile.js

I don't think this is universally automatable because you'd need to mandate so much structure, and Bower's ethos doesn't include mandating project structure. But adding transpiling and publishing to people's existing workflows is not hard.

Answer 29 · 2014-11-23T14:15:53.000Z

@ef4 has been using Github assets for releases. He might have useful insight int how this has gone/how he's automated the process: https://github.com/ef4/liquid-fire/releases

Answer 30 · 2014-11-23T15:18:38.000Z

It's been easy and painless. Though there's really nothing especially github-ish required, I could just as easily be pushing them to AWS and it wouldn't change my workflow. In my case the packages are found and downloaded manually, so Github is good for discovery.

Answer 31 · 2014-11-23T16:22:39.000Z

Having ran into all the problems @trek described myself, I now believe that ES6 Module Loader and ES6 modules are the solution. Luckily, we have a polyfill for the ES6 Module Loader which you can use today, but few people understand how to integrate it into their workflows.

That's where SystemJS comes in. SystemJS is a module loader implementation of the ES6 Module Loader spec (AFAIU) that allows you to import AMD, CJS, UMD, globals, and ES6 modules. Furthermore, you can author in any of those module formats. For the past 6 months I've been authoring ES6 modules (but you don't have to) and importing (using ES6 import) a multitude of module formats with no problems.

This is how far ES6 modules and the corresponding Module Loader spec will get us. It solves the problem of module interoperability.

At the next layer, you need to get a hold of modules. You could use the solution described above with Bower, but it would be very painful:

import foo from 'bower_comonents/foo/index.js`;

When an import is detected inside of this file, how will the module loader know where to find the requested module? In RequireJS, traditionally you had to manually supply some configuration to map module IDs to their paths on the file system. This sucked when you had nested dependencies. If I'm using module foo which depends on bar, why should I – the consumer – care about where bar is on my file system? Bower will download transitive dependencies, but that's all it will do. It will not wire dependencies together with the necessary map configuration.

@guybedford, who built the ES6 Module Loader polyfill + SystemJS, has been working hard to solve this problem. jspm is a package manager, so like with all other package managers you can do:

jspm install foo

But more importantly, jspm is aware of SystemJS. It writes the map configuration for you, so you can request modules by their ID instead of paths. Transitive dependencies will just work.

import foo from 'foo';

This is the best solution we have today to two of the biggest pain points when consuming packages as dependencies: module interoperability (e.g.. an AMD module consuming CJS) and tedious, manual map config. jspm also has a rich system for creating bundles.

We can start authoring ES6 modules today, but we can't ignore the fact that most libraries are using some non-standard module format. jspm is a solution that encompasses the whole ecosystem as it stands today, whilst building on top of future primitives.

(There are many more great things about the jspm + SystemJS + ES6 Module Loader workflow that I have missed out here. http://jspm.io/ is a good introduction.)

/cc @theefer @jamespamplin @guybedford

Answer 32 · 2014-11-23T18:22:58.000Z

For me

Those are the key words.

From others "I don't see what you're talking about, browserify solves this" is what you'll hear. Or "what? How come you don't just use Webpack. It solved this a long time ago".

I totally get that some consumers have found ways to work around this problem.

Answer 33 · 2014-11-23T18:25:18.000Z

How have we made it this far in this thread without the canonical reference being posted, I do not know:

Answer 34 · 2014-11-23T18:27:30.000Z

jspm is a solution that encompasses the whole ecosystem as it stands today, whilst building on top of future primitives.

It's not just a workaround. ES6 modules is the solution, but it uses ES6 Module Loader to fill in the gaps so you can work with what we have today. And to my mind, it's the only solution that's embracing standards at all.

webpack, however, doesn't build on ES6 primitives. It is just another hack.

Answer 35 · 2014-11-23T18:31:31.000Z

@trek I have removed those keywords. It will work for everyone.

Answer 36 · 2014-11-24T14:24:55.000Z

It's worth noting that jspm is typically used to install packages from github or npm, and it merely acts as a helper to install, configure and optionally bundle packages and their dependencies.

Under the hood, systemjs acts as a compatibility layer to load modules using any syntax.

The way jspm lets you bundle the whole dependency tree into one or or multiple files seems to be exactly what @trek describes when he says:

They're probably most interested in tree-shaking and whole-app minification rather than having dependancies pre-minimized for you. Minification is a fickle process, but I've noticed that delaying minification as late in the build process as possible often leads to smaller builds

@trek your original blog post seemed to almost exactly describe jspm (apart from the idea of all packages distributing "dists" in all formats). Did you consider it or know about it? How does a jspm solution compare with what you have in mind?

Answer 37 · 2014-11-24T15:46:42.000Z

@trek your original blog post seemed to almost exactly describe jspm (apart from the idea of all packages distributing "dists" in all formats). Did you consider it or know about it? How does a jspm solution compare with what you have in mind?

I'm very familiar with jspm, yes.

Like all consumer facing solutions (webpack, browserify, etc) jspm works extremely well for those consumers who choose to opt in. Not everyone wants to (or can) craft their entire app structure around a specific build tool or module process. If this was the case, we'd all move to browserify! This is especially true for existing applications.

This proposal is author focused. It's designed to provide official structure for authors to multi-format publish. Most of the top imported libraries are already published in many module formats but the ad-hoc solutions we've come up with means they can't be consumed automatically in a reliable way. A change to multi-publish means they could be automatically consumed by any number of consumer workflows.

The way jspm lets you bundle the whole dependency tree into one or or multiple files seems to be exactly what @trek describes when he says:

It is not really. Tree-shaking is not about shipping specific libraries in bundles. It's about automatically removing unused code

Answer 38 · 2014-11-25T17:42:43.000Z

I would also like Bower to try to solve the challenges of sharing libraries in an automatically consumable way. SystemJS / JSPM are not solutions for this because not everyone uses SystemJS / JSPM (I'm a user of SystemJS, I based StealJS off of it). For example, many people use RequireJS, or Browserify. I want to be able to share my code with them and have them consume it automatically.

For this to happen, it should be possible to write a RequireJS or Browserify plugin that can read the bower config and "go get" the right file.

As a micro alternative to the more comprehensive solution @trek proposed, I've thought about a main object where the keys are format types:

"main": {
  "amd" : "amd/foo.js",
  "cjs"  : "cjs/main.js",
  "es6" : "es6/foo.js",
  "global" : "standalone/foo.js"
}

This would simply point to different files based on what module loader was being used.
A RequireJS plugin can fetch this config in its locate hook and follow the "amd" path to bower_components/foo/amd/foo.js.

As I library author, I'm fine with exporting to every format type.

This would not have many of the benefits of @trek's proposal such as:

Authors can remove built code from source control.

which would be awesome. However, I care most about sharing code in an automatically consumable way to everyone.

Answer 39 · 2014-12-21T17:08:45.000Z

I've done some research whether it's feasible to unify this packaging format.

It's somewhat related to this issue, so: #1633

Answer 40 · 2015-09-07T22:11:39.000Z

This feature can be implemented using Pluggable Resolver (including building packages on-the-fly and downloading them prebuilt in custom location). The plan is to deprecate native resolvers and replace them with pluggable ones. I hope to make one of such pluggable resolvers a default in Bower 2.0.

I hope some of you will find time to experiment with it.

Thank you all for fruitful discussion!