"scoped" packages
isaacs opened this issue · 116 comments
The biggest request from enterprise users is a way to manage private modules with the same level of simplicity and ease as open source modules in the public npm registry. (A close second is group/role based management, but let's leave that for a subsequent discussion.)
Current State of the Art: Private Registry Replicas, Private Git, Proxies
Since the registry is (mostly) a Couch App, and we expose a public CouchDB interface to it, you can replicate the metadata into a database on your network, and optionally also use the npm-fullfat-registry module to fetch binaries into your copy of the db as attachments.
However, this is suboptimal in the following ways:
- You end up replicating hundreds of GB of stuff that you don't need.
- CouchDB with half a million binary attachments is not an easily maintained beast.
PayPal wrote kappa to deal with this, which doesn't require using a CouchDB full of attachments. There are several other similar proxy implementations that store the metadata and binaries to disk, but this is the one that I hear of the most often. This is a bit better than managing a Couch App, and has pluggable interfaces for caching, whitelists, etc.
Another option is to use private git repositories, either managed in-house or on GitHub. The npm client has some snazzy shorthands for GitHub repos, so you can do "foo":"org/node-foo" in your package.json dependencies, or npm install org/node-foo on the cli. However, there is a bit of an impedance mismatch and occasional friction in using git repos as npm dependencies. It's ok in a pinch, or if you have really well-established practices, but it's generally always going to be a bit of a mismatch, simply because git and npm make different assumptions about the world.
Name Conflicts
If you use basically any private system, you'll end up with name conflicts. Let's say our company creates a module called marked, to track places that we've marked in our location tracker app. Then we decide to use some public module that depends on @chjj's marked module, and it ends up installing our location tracker doohickey as a dep, because the registry is set to npm.my-company.com.
We can get around this by implementing some rather convoluted logic to manage multiple registries, and keeping track of where on the list of registries we started in the lookup process, and always go down the list rather than up.
However, this means that we can't do things like fork the markdown parser package named marked to fix a bug, and have our dependency lookups find that instead. There is no way to solve both problems, because they contradict one another.
What's worse, the logic for knowing how to resolve conflicts gets really confusing to reason about, and when you're looking at code, it's not clear what require("marked") refers to.
So, we have to be very careful to not use names that are publicly used, or squat on public names, which disrupts the OSS community flow, and so is generally bad behavior.
Namespace Constraints
It'd be nice if we could scope modules to a specific user or organization. If such scoped modules were able to be made private, and if they could be unambiguously recognized by the client, then it would know that it might have to do a "give me access to this private thing" authorization dance in those cases.
Some constraints:
- There must be a simple and unambiguous way to say "install
mycompany'smarkedmodule". - It must not conflict with any currently allowed names. (Ie, it should be invalid in the current paradigm. Not hard to do, since names are very restrictive.)
- It should not conflict with the
user/reposhorthand used for GitHub. (Oops. That would've been a really nice way to go. Oh well, backwards compatibility. It sucks, but not as much as it would suck to break users.) - If you do
npm install <scoped module>then you should be able to dorequire("<scoped module>")in your JavaScript, and view the data athttps://registry.npmjs.org/<scoped module>in a web browser. - Since (3) implies we have some kind of magic character or syntax, (4) means that any such magic character/prefix/syntax must not use characters that are magical in URLs, file systems, or shells. (For example,
npm install ~org/fooornpm install $org/foowould be fine in the fs and url, but the shell will expand~orgtothe home of user organd$orgtothe value of the org var.+and#and%all have similar problems in other places.)
I've discussed this with the people here at npm, Inc., and with a few of the most prolific npm contributors and publishers, and users at several large companies. The scheme that seems to be the most intuitive, which also fits within the rather narrow constraints is @org/pkgname.
Multiple Registries
Each @org will map to a specific registry in the config (or the default --registry config, which defaults to registry.npmjs.org, if not specified).
We can then also scope auth info to each registry, and voila, we have multiple registry support that works, doesn't conflict with anything that already exists, doesn't result in fetching the right thing from the wrong place.
@org/pkg modules will be dropped onto the filesystem at node_modules/@org/foo, so require("@org/pkg") will work.
And since that's the actual package name, you'll do "dependencies":{"@org/pkg":"1.x"} in your package.json file.
As a way of implementation details, the @org/pkg thing will map to npm://{registry}/@org/pkg, where {registry} is the registry that is assigned to the @org or the default --registry config. npm:// urls will be fetched via https, and always send authentication information, and treat 404's as a potential "maybe you don't have permission" response.
This does mean that the registry will probably not be strictly just a Couch App for much longer. In practice, it already isn't, but since CouchDB chokes on doc names that have a / in them means that we'll have to %2F it, and that's just a bit confusing. Maybe we could use - instead of /, but the niceness of / as a filesystem separator is really awesome.
We'll still keep shipping an OSS reference implementation, but it probably won't do private modules, since that usually is going to require some kind of custom logic. Basically, npm-registry-couchapp will be just a part of the registry implementation, instead of effectively the entire thing. In addition, there'll be a minimal "front door" bit that proxies to the Couch App, manages roles, restricts GETs, etc., and that's where you'll find hooks for auth magic. That also means we can avoid putting the attachments into CouchDB in the first place, since all we do is then immediately take them out of CouchDB, without breaking old clients, and opens the door to doing other creative stuff, like more efficient views, relational reporting stuff, and putting download counts and other analytics right into the data itself. Handwavey vaporware, etc.
Feedback please
Would you use this? What would you like it to do? Anything about it that you think would be weird or confusing? Is there an approach you'd like better that meets all the constraints?
+1
Questions
- is the
"name"in the package.json"@org/foo"? - how do we deal with
orgnames in the public registry ? How do we reserve them ? For example can I publish"@npm/config"? What if I have an internal"@foo/bar"and want to use a public module that's also called"@foo/bar" - are we updating shrinkwrap to know how to deal with
"@org/name"? as far as shrinkwrap cares there is just a folder called"@org"in node_modules with no package.json. - what's the user experience when you accidentally
require("@org") - will the
node_modules/@orgfolder need a package.json ? What will blow up if the folder doesn't have a package.json
First class citizen questions
The existing story around private git links has some usability problems.
- will
npm docs @org/foowork, it doesn't for git links - will
npm info @org/foowork - will npm be able to dedupe these @org/foo modules properly, it can't dedupe git links
- will
npm update @org/foowork ? It just installs the latest for git links - will
npm i @org/foo@{range}work ? it doesn't for git links - will
npm link @org/foowork ? it doesn't for git links.
We should think of all the areas where git+ssh links are a second class citizen in terms of the features of the npm cli and ensure that @org/foo does not suffer similar problems.
This will help address
manage private modules with the same level of simplicity and ease as open source modules
By making @org modules just as easy & simple as open source ones.
btw this sounds great :) 👍 on this, let's make it really really smooth & easy to work with.
is the "name" in the package.json "@org/foo" ?
Yes.
how do we deal with org names in the public registry ? How do we reserve them ?
Effectively, orgs are users. You've already "reserved" @raynos, for example.
Names will be handled similar to the "disputes" policies that we already have. If your company name is foo, and some jerkwad is squatting on @foo and not using it, we may be glad to evict them, but that's a "have the humans use their human brains for moderation" kind of issue.
For example can I publish "@npm/config" ?
No, because you're not in the @npm org.
What if I have an internal "@foo/bar" and want to use a public module that's also called "@foo/bar"
Don't do that :)
You do bring up a reasonable issue, in that this doesn't solve all namespace problems, but "merely" adds a single additional dimension, so there's "only" one less order of namespace collision magnitude.
are we updating shrinkwrap to know how to deal with "@org/name" ? as far as shrinkwrap cares there is just a folder called "@org" in node_modules with no package.json.
Yes. Shrinkwrap will know how to deal with @org/pkg as the name of a package.
what's the user experience when you accidentally require("@org")
Probably it will throw a "module not found" flavor of ENOENT.
will the node_modules/@org folder need a package.json ? What will blow up if the folder doesn't have a package.json
Nope. node_modules/@org will probably just have folders in it. If we do throw some kind of helpful metadata in there, it'll probably just be for the npm client's use.
The existing story around private git links has some usability problems.
Tell me about it! They're kind of awful.
will npm docs @org/foo work, it doesn't for git links
Yes.
will npm info @org/foo work
Yes.
will npm be able to dedupe these @org/foo modules properly, it can't dedupe git links
Yes.
will npm update @org/foo work ? It just installs the latest for git links
Yes.
will npm i @org/foo@{range} work ? it doesn't for git links
Yes.
will npm link @org/foo work ? it doesn't for git links.
Yes.
We should think of all the areas where git+ssh links are a second class citizen in terms of the features of the npm cli and ensure that @org/foo does not suffer similar problems.
Agreed. We're starting with the experience of non-org-scoped modules ("global" modules, in our internal parlance), and going from there. Rather than using Git as the model, since it is really klunky to work into npm's way of doing things.
This will help address
manage private modules with the same level of simplicity and ease as open source modules
By making @org modules just as easy & simple as open source ones.
That is the plan :)
Another thing:
"Global" modules (ie, unscoped, what we have now) will not be allowed to depend on scoped modules. This is important.
Otherwise, the community as it is now would be unduly harmed, because we'd very quickly end up with global packages that can't be installed by old versions of npm.
This "namespaced maybe-private org-scoped" thing is only going to work with newer npm clients of the future, but there's no reason to invite ruin on older npm clients that are currently working, unless it's actually necessary for some reason.
I want people to upgrade to the latest and greatest npm, but it should be much more carrot than stick.
just for clarification, if an @my-org is scoped to a private registry https://registry.my-org.com will packages be fetched at https://registry.my-org.com/@my-org/my-module or just https://registry.my-org.com/my-module
if it were the latter, more existing setups may be supported, and if you have a private registry that supports the new @org pattern, you could set the url for for your orgs modules to https://registry.my-org.com/@my-org
@hayes That's interesting. Our assumption had been that it'd be https://registry.my-org.com/@my-org/my-module, so that the npm client could just Do The Same Thing no matter where it's fetching stuff from. But, I can definitely see the benefits there. Maybe "send the org name" could be configurable in the same way that the registry is, for maximum flexibility?
I am a big fan of flexibility. I think that could be a very useful options. but after thinking about it for a minute I realized it couldn't be the default
@izs thanks for opening a place to discuss this.
Name Conflicts
This is of course inevitable, which is why the proxy I wrote with @jcrugzz has a policy format which understands both blacklists and what we refer to as "known private modules". This also solves the question from @Raynos
What if I have an internal "@foo/bar" and want to use a public module that's also called "@foo/bar"
Because you could simply blacklist your @foo/bar. The default for all publishes is to proxy them to your private registry, which leads to the one caveat of our private npm product: existing public modules proxy pass through just fine, but that new public modules have to explicitly be published directly to the public registry. This gotcha does come up every now and again, I've seen support tickets heard this from 4-5 customers (about 1-3%).
Namespace Constraints
I agree in principle that we need something to hang users off of. Discussions around "groups" have happened on and off for years, which is essentially the same thing as what is being referred to "orgs" now. What I don't explicitly see the value in is the @ scoping. If you are implementing private packages using the existing "private" then a large breaking change is already being introduced since up until now "private" has meant "never publish this thing to any registry ever".
Knowing that private packages will in principle cause a large swath of breaking changes for users why not just roll prefix-less orgs into npm@2.0.0 and start deprecating the Github syntax now? It should be pretty easy to run a view to see how many packages are using that now. Whichever thing has lower impact from data should be the answer. I'm separately of the opinion that backwards compatibility shouldn't be preserved just because reasons, but then again seeing @ry this week might've just made me nostalgic :)
Multiple registries
We'll still keep shipping an OSS reference implementation, but it probably won't do private modules, since that usually is going to require some kind of custom logic.
That's a pretty bold statement, but we of course have our own pieces that are private at Nodejitsu so I can't call the kettle black. The biggest request I would make it make importing and exporting private modules out of the private thing easy because both of our customers will want that.
+1 to moving more complex logic out of the CouchApp as jcrugzz outlined in his post back in February. With that in mind, if you are overhauling the auth system we should really consider implementing OAuth2 as a proper way to grant authorization to other apps. It's not the best, but OAuth2 has it's fingers in enough things that it's a sane first choice.
If you move the main proxy logic out of the CouchApp (i.e. by removing the rewrites) then the %2Fs aren't too bad. I've done this a lot for other CouchDB based applications to get vanity URLs and just taking advantage of the kv + map-reduce qualities of CouchDB. Clearly I have other opinions about where binary data should be, but that's outside the scope of this discussion.
Aside: Package signing
If we are really going to say "send all requests to @org/* packages to http://etc-etc-etc" we still have not prevented the specter of MITM attacks because by introducting N places to get packages we simply have N more places that could be vulnerable. I think that "orgs", "teams" (i.e. group/role management), and package signing are very much intertwined and should be thought of as a single unit to avoid painting ourselves into a corner as we unfortunately have done here with Github syntax.
OK but what does this have to do with twitter?
but there's no reason to invite ruin on older npm clients that are currently working
This argument is void. The introduction of "foo": "^1.0.0" already broke old npm clients. The community was not ruined by this change as far as I can tell.
will the public registry host packages that specify a specific @org in its dependency to be a different registry? (are those settings in the package.json, or are they npm client settings)
What's the use case for this? Is it just specifically for a particular company to host their secret sauce and never give anyone else access to it? Will they migrate them to the global namespace if they ever want to make those modules public?
@Raynos An argument to disallow publishing packages that depend on @org packages to the public registry is that, since the binding between @org -> url is local to each machine, packages published with @org deps would be unresolvable by clients who did not share .npmrcs, no matter what version of the npm client they had installed (if I'm understanding this correctly).
How will publish rights to a module at "@org/foo" work ?
Currently I can just npm owner add mattesch @org/foo ? will the user mattesch then be able to publish to a module at @org/foo ?
To clarify some confusion, where does the config mapping between @org and registry.my-org.com live?
- The public registry?
- A client's
.npmrc? - A scoped client
.npmrc? (i.e.,~/.npm/@foo/npmrc?) - In the package.json?
And does a missing @org cause an error, or default to the public registry? (from re-reading, I assume "it defaults to public registry", but want to be clear)
@chrisdickinson The resolution of an @org to a given registry happens in the client, although the responsibility for validating permissions will probably be scattered throughout the infrastructure. All configuration will be handled using the configuration mechanisms that exist today, although if / once support for multiple / per-project configuration files lands, it might make sense to migrate some of this there. The implication is that @org can map to different registries for different users, depending on their configuration. We have discussed how to consolidate these mappings site-wide for large installations, although we haven't arrived at a final decision on how that's going to work.
Something useful to keep in mind is that the mechanics of resolving packages exists independently from the permissions governing how the server determines whether a given publish is permissible (to speak to @Raynos's point). I'll let Isaac speak to the specifics of how that will work.
@othiym23 Thanks! The points I was somewhat foggy about were as follows:
- Whether
@orgwould exist as an entity outside of private registries, based on @isaacs's comment, "Effectively, orgs are users. You've already "reserved" @Raynos, for example." - If the above is true, whether they would become "the future" of packages published on npm -- i.e., would it become passé to public an unscoped package?
- Can dependencies flow from public npm into
@org/packages(looks like "no, due to client compatibility", but there might be other concerns there). - Can dependencies flow from
@a/xto@b/y(one package in one org depending on another package from another org)?
As I understand it, the @org syntax is kind of like the Host header combined with /etc/hosts -- it gives clients an opportunity to use a different registry, but if not given, it will request whatever the mainline --registry is, while letting that registry know that it wants to request from a certain @org. Given that, my assumption would be that "unscoped modules" would remain the norm, "unscoped public modules" could not depend on @org modules (due to the risk of depending on a private/unresolvable dependency), and @orgs could freely depend on public packages, packages from their own @org, and between different @orgs. Is this correct?
What I don't explicitly see the value in is the @ scoping.
Please see the list of namespace constraints in the OP.
If you are implementing private packages using the existing "private" then a large breaking change is already being introduced since up until now "private" has meant "never publish this thing to any registry ever".
This probably will not use the "private" field in package.json. If it does, it'll only be for new packages, since existing packages can never have a name like @org/name.
Knowing that private packages will in principle cause a large swath of breaking changes for users
Can you elaborate on which breaking changes are planned? I don't see any impact on existing users. They may be left behind if the bulk of development activities move toward namespaced modules, but their existing workflows will keep working.
why not just roll prefix-less orgs into npm@2.0.0 and start deprecating the Github syntax now?
Backwards compatibility.
I'm separately of the opinion that backwards compatibility shouldn't be preserved just because reasons, but then again seeing @ry this week might've just made me nostalgic :)
Noted :)
That's a pretty bold statement, but we of course have our own pieces that are private at Nodejitsu so I can't call the kettle black.
Sorry, I'm not sure which is the bold statement there? Most of what we do at npm, Inc. is developed in the open, and I don't see any reason to change that.
With that in mind, if you are overhauling the auth system we should really consider implementing OAuth2 as a proper way to grant authorization to other apps.
Xylophone. Implementing an OAuth2 provider for npmjs.org is an idea worth exploring, but it's well outside the scope of this discussion.
Aside: Package signing
Xylophone. Package signing is important, but outside the scope of this discussion.
Whether @org would exist as an entity outside of private registries, based on @isaacs's comment, "Effectively, orgs are users. You've already "reserved" @Raynos, for example."
On Github, the top-level namespace is the same for both users and organizations. Same deal here. This concept is not exclusive to private registries.
If the above is true, whether they would become "the future" of packages published on npm -- i.e., would it become passé to public an unscoped package?
Given the fact that private packages can depend on public packages, but not vice versa, this is not necessarily the future. As an aside, I find it easiest to think about @orgs as segmenting one namespace, rather than creating multiple top-level namespaces.
Can dependencies flow from public npm into @org/packages (looks like "no, due to client compatibility", but there might be other concerns there).
If you mean, can package @aoaioxxysz/euterpe depend on request, most assuredly. The other way, no.
Can dependencies flow from @a/x to @b/y (one package in one org depending on another package from another org)?
Yes.
As I understand it, the @org syntax is kind of like the Host header combined with /etc/hosts -- it gives clients an opportunity to use a different registry, but if not given, it will request whatever the mainline --registry is, while letting that registry know that it wants to request from a certain @org. Given that, my assumption would be that "unscoped modules" would remain the norm, "unscoped public modules" could not depend on @org modules (due to the risk of depending on a private/unresolvable dependency), and @orgs could freely depend on public packages, packages from their own @org, and between different @orgs. Is this correct?
This is correct.
The introduction of "foo": "^1.0.0" already broke old npm clients. The community was not ruined by this change as far as I can tell.
We only switched to ^ as a default once it had been supported in stable releases of Node/npm for almost a year, iirc, and in fact, there was quite a bit of wailing and gnashing of teeth, leading eventually to the recent addition of the --save-prefix config.
The community wasn't ruined, but it was also a much smaller change than this addition, and in retrospect, I would have liked very much to have put more thought into avoiding that issue. In this case, the rule that "global modules can't depend on scoped modules" avoids the problems rather nicely.
How will publish rights to a module at "@org/foo" work ?
Still working out the best way to accomplish that, but yes, you'll add a user or a group to the package, and then there can be a way to add a user to a specific group.
Part of the purpose of our on-prem beta program is to give us a little petri dish in which to experiment with a few different ways to manage auth and groups. Most likely, there'll be group management by the end of the year in the public registry, and we can have another one of these to discuss that. Until then: xylophone.
@thomblake
What's the use case for this? Is it just specifically for a particular company to host their secret sauce and never give anyone else access to it? Will they migrate them to the global namespace if they ever want to make those modules public?
Scoped modules will be either public or private, at the whim of the owner. I will not expect that they will ever be migrated to the global namespace. This thus satisfies another common request: having simpler/shorter names for packages, scoped to a specific owner, so that you can have require("@company/foo") instead of require("company-foo-but-not-the-foo-for-bar-the-other-for-the-one-for-baz-you-know-for-kids").
I think you're understanding it correctly. Deps can go from scoped -> scoped, scoped -> global, or global -> global, but never global -> scoped.
One downside of the @-sign that I didn't realize until posting this discussion here: we've been spamming Olly Groves (https://github.com/org) quite a lot. I apologize for that, Olly. You may want to ignore this thread entirely, I suspect it will continue.
This thus satisfies another common request: having simpler/shorter names for packages, scoped to a specific owner, so that you can have require("@company/foo") instead of require("company-foo-but-not-the-foo-for-bar-the-other-for-the-one-for-baz-you-know-for-kids").
To amplify this, this makes it easier for people to fork and experiment with existing packages to scratch particular itches without the original package needing to change or grow unnecessarily.
Can namespaces mutate their privateness?
... to expand:
[16:57:11] <chrisdickinson> the other problem is the mutability of organization private-ness
[16:57:50] <othiym23> chrisdickinson: care to expand?
[16:58:16] <chrisdickinson> I have @chrisdickinson on npm. Folks start do depend on me from their @scoped packages
[16:58:40] <chrisdickinson> I make @chrisdickinson private
[16:58:48] <chrisdickinson> I've broken all of the @scoped packages.
EDIT: I misunderstood. @othiym23 straightened me out: @org/packages publicly available on registry.npmjs.org will always be public (modulo unpublish); going private means you've (individually) spun up a private registry for @org.
Since an organization can be depended on by other organizations, and because local configuration can change where @org resolves, we'll have to register an npmjs.org account for the organization to ensure we don't accidentally shadow some other organization.
If @org is directed to some other registry, but there are some @org packages that are public, the other registry would need to have both the private and the public packages, as the client won't fallback to the public registry.
Previous discussion within the community seemed to have concluded on scoping packages by registry, not by organization. I want to make sure I'm entirely clear of the impact here.
Turned into a bit of an essay, but please bear with me. I am not simply disagreeing for the sake of being difficult.
The private mechanism is important here
Can you elaborate on which breaking changes are planned? I don't see any impact on existing users. They may be left behind if the bulk of development activities move toward namespaced modules, but their existing workflows will keep working.
So if the TL;DR here is "org ns as step towards private modules" it is reasonable to ask what the mechanism for making things private is going to be.
- Is it going to be a flag in the
package.json? - Is it going to be done through the
npmCLI? - Is it going to be done through
npm-www?
I (wrongly) assumed that the plan was to repurpose { "private": true } in package.json files which was the breaking change I was referring to.
Why so in-favor of strict backwards compatibility?
why not just roll prefix-less orgs into npm@2.0.0 and start deprecating the Github syntax now?
Backwards compatibility.
Right now there are 225 public packages which use this syntax in their dependencies (did not check *Depedencies like devDependencies), or roughly 0.320% of all packages (source). So that is not a lot, but it is enough to want to find a backwards compatible way to fix this. That mechanism doesn't have to be as strict as to introduce a new syntax that every new node developer is going to have to learn. What about this flow:
LOAD_AS_ORG(X,Y)
1. Check to see if `X/Y` exists in npm.
2. If not, assume Github repo with `X/Y` and warn the user of deprecation.
It would require formal deprecation of the Github syntax, but it precludes a new syntax. Further, we could use the new @ syntax to replace the existing Github syntax so people can switch when they see the deprecation notice. This is clearly not the list of all consumers of this feature, but it is representatively small. Any interest in surveying users on their existing use of this feature? Have you already?
Based on the anecdotal evidence I've seen organizations would be used 1-2 orders of magnitude more frequently than the existing github shorthand -- most of them use the git:// URL syntax explicitly. So if we could deprecate and replace that seems like a win, not a loss.
Why are we complicating things?
@othiym23 @chrisdickinson @izs
I think you're understanding it correctly. Deps can go from scoped -> scoped, scoped -> global, or global -> global, but never global -> scoped.
I misunderstood. @othiym23 straightened me out: @org/packages publicly available on registry.npmjs.org will always be public (modulo unpublish); going private means you've (individually) spun up a private registry for @org.
If scoped modules are public by default as @chrisdickinson and @othiym23 said, why the restriction? There is a widely used workflow at companies using Node.js (and even client-side JS) which implicitly conflicts with it:
- Company X makes a module,
some-thingthat they intend to Open Source. - Company X publishes that thing to npm as a private package
@company/some-thing - A year later Company X wants to release
some-thingas Open Source, but the globalsome-thingis taken. - Company X is now discouraged from Open Sourcing because of additional breaking config changes in
package.jsonfiles if they change the name tosome-other-thingand making the scoped@company/some-thingOpen Source can't be used by global modules.
Xylophone or just not on this issue?
How will publish rights to a module at "@org/foo" work ?
Still working out the best way to accomplish that, but yes, you'll add a user or a group to the package, and then there can be a way to add a user to a specific group.
Why is this a xylophone? It is hard (if not impossible) to completely assess the intrinsic value of the proposed organizations feature without fully understanding how the publish and user management mechanics will work. e.g. our customers love user sync across their team.
The same goes for OAuth2 and package signing. All of these features are intertwined in nuanced ways: the implementation details matter here. I can move that part of the discussion to another issue if you'd prefer.
Can we count on this module.js "hack"?
So this thing is a nit, but it seems we got lucky with the @org syntax, because according to the docs it should not exist since the @org directory does not need a package.json file.
LOAD_AS_DIRECTORY(X)
1. If X/package.json is a file,
a. Parse X/package.json, and look for "main" field.
b. let M = X + (json main field)
c. LOAD_AS_FILE(M)
2. If X/index.js is a file, load X/index.js as JavaScript text. STOP
3. If X/index.node is a file, load X/index.node as binary addon. STOP
//
// ... ? We are here, in undefined territory.
//
module.js in core hasn't changed in almost six months, but I know that bundling into a single executable was on @tjfontaine's radar for node@0.14 which will require changes to the module loader. Summoning him and @bmeck here to see if that has any bearing on this discussion.
What is a "reference implementation"?
We'll still keep shipping an OSS reference implementation, but it probably won't do private modules, since that usually is going to require some kind of custom logic.
Sorry, I'm not sure which is the bold statement there? Most of what we do at npm, Inc. is developed in the open, and I don't see any reason to change that.
What's bold is that you're saying the OSS npm registry is going to become a "reference implementation", which won't be run in production by npm Inc. Please forgive me if I am misinterpreting this, but I've always read the phrase "reference implementation" in the past as "a toy for hobbyists". Is that the intention here?
Part of the purpose of our on-prem beta program is to give us a little petri dish in which to experiment with a few different ways to manage auth and groups. Most likely, there'll be group management by the end of the year in the public registry, and we can have another one of these to discuss that.
Could you clarify the above? Is there an implementation of these features somewhere that is not yet public? Sorry for prying, but your initial segmentation about what would be private and what would be public was rather nebulous at best and I know I'm not the only one who would like a clearer line drawn in the sand.
Just to clarify, people want to change require to include a scope mechanism as well? This would be pretty major since it could break node_modules as a folder if you do have 2 modules with the same name but different scopes.
If we do use scope in require I would avoid using a specialized string in require. Keeping it separated ala require.scope(org)(module_name) or require.scoped(org, module_name) (idc syntax, just separated) since collisions can still occur if only module_name is present unless we want to continuously add new string syntax to require.
Either way, I would need more info on what is actually being proposed to be put onto disk to say much more. This would not interfere with archives or bundling most likely since that just reuses the require mechanisms.
Based on the anecdotal evidence I've seen organizations would be used 1-2 orders of magnitude more frequently than the existing github shorthand -- most of them use the git:// URL syntax explicitly. So if we could deprecate and replace that seems like a win, not a loss.
Remember that npm i flatiron/winston --save expands the github uri to a git link and stores the git link in the package.json.
I believe npm install is far more common then manually writing git links in package.json
I don't know actual numbers though
Based on the anecdotal evidence I've seen organizations would be used 1-2 orders of magnitude more frequently than the existing github shorthand -- most of them use the git:// URL syntax explicitly. So if we could deprecate and replace that seems like a win, not a loss.
Remember that
npm i flatiron/winston --saveexpands the github uri to a git link and stores the git link in the package.json.I believe
npm installis far more common then manually writing git links in package.json
This I did not know. Thank you for the clarification. Digging into that further the most important reason for backwards compatibility should be changing the behavior of packages already published to the npm registry.
The flow I suggested falling back to git:// URLs works even better here because there is nothing to be backwards compatible with. We simply need to make consumers of the CLI aware of the change. Crunching the numbers for usage of git:// URLs in packages published would be less representative since they could have been added explicitly without the CLI syntax you mentioned.
@bmeck the syntax being proposed here works as is today which was a bit of a head turner for me. e.g. consider:
module/
index.js
node_modules/
@org/
winston
package.json
where your index.js is:
var winston = require('@org/winston');
winston.info('wat');The above will just work. I had to verify it to believe it since it isn't documented officially. Any bundling should consider these folders or use Module._resolveFilename and/or Module._resolveLookupPaths when traversing require statements. Probably xylophone though to stay on the meat of the discussion here.
re: complicating things.
If scoped modules are public by default as @chrisdickinson and @othiym23 said, why the restriction?
My primary issue with allowing pkgA depends-on @org/pkgB is the potential for the client (given this scheme) to be talking about a different @org/pkgB than is available on the target default registry -- or referring to one that is not present on the target default registry. This opens up the public namespace to pollution with packages that either cannot successfully install or do not work correctly upon installation. Disallowing reentrance strengthens the assurance that the packages in the public namespace are installable (disregarding existing git:// links to private repos / broken http://tar.gz deps).
There is a widely used workflow at companies using Node.js (and even client-side JS) which implicitly conflicts with it: [...]
It also puts up guard rails keeping users from bad practices in this regard: if A depends-on B depends-on C, open source C, then B, then A -- not B first. Put another way, it keeps users from accidentally making public a package which cannot be installed by any user outside of their company.
re: the privacy mechanism:
So if the TL;DR here is "org ns as step towards private modules" it is reasonable to ask what the mechanism for making things private is going to be.
Is it going to be a flag in the package.json?
Is it going to be done through the npm CLI?
Is it going to be done through npm-www?
As I understand it (and to recap and make sure we're on the same page):
- Namespaces are a flex point for the npm client to override what registry it is referring to.
- Dependencies can only flow into them and between them -- non namespaced packages can never depend on a namespaced package.
- Clients may override the mapping between namespace & registry url. Find $REGISTRY:
- If the registry url is overridden locally [*]:
- ... Use the overridden registry url.
- Otherwise,
- Use
--registry=<url>, if present. - Use
NPM_REGISTRYenv var, if present. - Use
.npmrc'sregistrysetting, if present. - ... Eventually fall back to
registry.npmjs.org.
- Use
- If the registry url is overridden locally [*]:
- Packages are resolved at $REGISTRY using
name/version, wherenameis the full name of the dep (including the@org/). Allow $REGISTRY to do any special resolution based on@org/.
This leads me to believe that the privacy setting is the overridden @org -> registry.url mapping on the client (which might increase the risk of accidental publishes?)
Alternatively, it's the combination of the client's org/url mapping and the default registry's ACLs for that @org -- i.e., does user X have access to publish package Y on @org Z? Since packages are always requested from the server with @org/ included, the server has a chance to make its own determinations there.
Personally, I'd be much more comfortable with the alternative situation -- being able to flip a switch on registry.npmjs.org and say "nope, no public packages from @urbanairship please" rather than the (error-prone) process of making sure every developer always has their npmrc correctly configured to prevent accidental publishing.
[*]: Leaving this vague as I'm not sure if it's going to be .npm/@org/npmrc or NPM_REGISTRY_org or ... really at all how that mapping will be stored.
What if we want to add a shortcut for Bitbucket-repos?
What about this Syntax:
npm install github user/repo
npm install bitbucket user/repo
npm install organization user/module
Then throw them in like this:
node_modules/
module1
module2
@github/
user/
repo
@bitbucket/
user/
repo
@organization/
user/
module
if you are interested in providing feedback on this issue you may also be interested in feedback on https://normalize.github.io/ (https://github.com/normalize/discussions/issues/3)
I like @org/package much more than org/package, independent of any backward-compat issues. The @ sign is a nice differentiator. E.g. require("@kriskowal/q") versus require("q/queue"): the former is Kris's Q package; the latter is the queue module inside the global q package.
So I'm a little surprised you're spending so much time arguing for org/name, @indexzero. It doesn't really have any upsides besides one less character to type.
re: complicating things
Your technical argument has merits, but you did not respond to the cultural or incentive argument. The community goal has always been (as I understood it) "don't slow down growth". Adding extra steps to the publish ceremony water dance that developers must do just because they work at a company and want take advantage of this feature will discourage them from using it.
The fallback mechanism that allows for global packages to depend on scoped packages is actually quite simple:
INSTALL_SCOPED_MODULE(org, module)
1. Does REGISTRY configuration exist for "org"?
- YES: Attempt to fetch "org/module" from that registry
- 200 OK. STOP. Module installed.
- 404 Not found. Attempt to fetch "org/module" from "default" registry or registry.npmjs.org
- 200 OK. STOP. Module installed.
- 404 Not found. STOP. Module not found.
- NO: Attempt to fetch "org/module" from "default" registry or registry.npmjs.org
- 200 OK. STOP. Module installed.
- 404 Not found. STOP. Module not found.
It also puts up guard rails keeping users from bad practices in this regard: if
A depends-on B depends-on C, open source C, then B, then A -- not B first. Put another way, it keeps users from accidentally making public a package which cannot be installed by any user outside of their company.
This is where I'm getting really confused, especially w.r.t. to multiple registries. Consider this scenario:
- I am a complete n00b
- I think up a name for an org, like @imanawesomenoob
- I run
cd /iamanawesomenoob
cat package.json | grep name
"name": "@iamanawesomenoob/some-module"
npm publish
Where does it live? And more importantly: where does it live it I decide to start making it private? Does it only exist privately if I run my own private registry? Or is there a place where it is hosted?
re: privacy mechanism
Personally, I'd be much more comfortable with the alternative situation -- being able to flip a switch on registry.npmjs.org and say "nope, no public packages from @urbanairship please" rather than the (error-prone) process of making sure every developer always has their npmrc correctly configured to prevent accidental publishing.
Yes. I agree with this as well, which is why our private npm product default all publishes to private to avoid this behavior. As wonderful and amazing and fulfilling as writing Open Source is, I believe that the number of private modules dwarfs the number public modules by at least one order of magnitude. This anecdotally makes sense when you consider that every application is, in fact, a module. So in the "paid and corporate-y" world, the scenario of accidentally publishing a private module public is more likely than accidentally publishing a public module private. Optimizing for this fact would cause much rejoicing of future developers.
re: scoped modules vs. sub-modules
I like
@org/packagemuch more thanorg/package, independent of any backward-compat issues. The@sign is a nice differentiator. E.g.require("@kriskowal/q")versusrequire("q/queue"): the former is Kris's Q package; the latter is thequeuemodule inside the globalqpackage.
That is a very good point that I simply had not thought about. So good it should be in the list of "Namespace constraints" in the OP:
- It should be easily differentiable from "submodule syntax". That is,
require("org/module")should be easy to understand when compared torequire("global-module/sub-module-path")
My original argument was counter-pointing the argument for backwards compatibility of a feature that did not appear to be used heavily. Your point makes that moot because disambiguation of what's being required is much more important than backwards compatibility.
Re "Reference Implementation" vs "Toy for Hobbyists"
A reference implementation is not necessarily low-quality. In fact, being a low-quality implementation can reduce its value as a reference!
Both production and reference implementations benefit by being highly performant and easy to use. However, for a reference implementation, "ease of use" is prioritized over "high performance", whereas in a production implementation, the opposite is the case.
The public npm registry has not been a single couchapp exposed to the world since fall of 2013. Today, it is a write-master, several read-only replicas, a CDN, and a static file server, with a bunch of various workers and rewrites and automation keeping it all together. The npm-registry-couchapp piece is important, but it's not as if the production implementation and reference implementation necessarily need to be identical.
A reference implementation is a specification that can answer questions you didn't think of ahead of time. I believe that empowering innovation has massive long-term upsides, even if some of that innovation might happen in the heads of potential competitors. So, we have decided to do this development in the open, including making the standalone reference implementation include more than just a couchapp. This means it might be a bit harder to set up and use, but it will also be a better reference. Tradeoffs.
My primary issue with allowing pkgA depends-on @org/pkgB is the potential for the client (given this scheme) to be talking about a different @org/pkgB than is available on the target default registry
While you're right that this is still a problem, it's a much smaller problem than the one we have today. An organization will likely have only one name, but many modules. Requiring that ALL of their modules don't conflict with anything public is much more hazardous than requiring that their organization name doesn't conflict with anything public. How many "paypal"s or "yahoo"s are there in the world?
Re require.scope("@org") or some such:
We won't be modifying node's module system. But, it might be possible to do something like this in a userland module. Node's module system is actually quite well documented, and everything proposed here works today. Try it!
I believe npm install is far more common then manually writing git links in package.json
Indeed, most people using this for private modules are not publishing them to the registry. Your website depends on company/foo and company/bar which are both private. You don't publish that to the npm registry.
I've seen a lot of stuff that depends on joyent/private-thing when I worked at Joyent. None of it was published to the registry, though.
Several people have brought up the problems with deprecating and re-purposing the GitHub shorthand syntax. That's not going to happen.
Re "path to dev private, open later":
The sequence will not be to work on @org/foo in private, and then publish it as foo.
The sequence will be to work on @org/foo as a private module, and then make @org/foo public.
Being scoped does not mean that a package is necessarily private. The organization owner will have the option to make packages public or restricted.
This has been repeated several times in this thread.
Many of the mechanisms for this are still TBD, and there may well be competing implementations.
The thing about "open development" is that you get to see stuff before all the questions are answered. One might say that's the only thing about open development, in fact! Expect some partly-baked experimentation.
In the interest of keeping this conversation useful for newcomers, and not them scaring away with an even more impenetrable wall of text, any further messages that reintroduce already-answered questions may be deleted or edited at the discretion of the npm project maintainers. Thanks.
The fallback mechanism that allows for global packages to depend on scoped packages is actually quite simple:
We're not going to send 404-ing requests to the main registry. Doing that would be an information disclosure vulnerability.
Where does it live? And more importantly: where does it live it I decide to start making it private? Does it only exist privately if I run my own private registry? Or is there a place where it is hosted?
It'd be published on the public npm registry. As mentioned above, the specific mechanisms for how that'll be handled is still TBD.
Scoped modules are "a step towards private modules and multiple registry management". They're not "a private module solution".
It may be that (org)name would work as well @org/name, and avoid discussions of those modules on github getting emailed to random users.
I think I missed how this will work with private (probably temporary) forks of a public module. Say you find a bug in the module wonderbar, so you github fork it to your organization, patch it, PR the author, and while waiting for the fix to land on npmjs.org, you publish it to your private registry, npm.mycompany.com.
You don't change its name, and don't want to (you have dozens of modules depending on wonderbar, and don't want to go through and hack up all their package.json, and then unhack them once the PR is accepted and wonderbar is republished, and anyhow, if you were going to do that you could just hack them to use github refs, like somecompany/wonderbar, but github refs suck for many reasons already discussed).
Currently, to make this work, you point your default registry at your private registry (registry.mycompany.com), and probably arrange to have the private reg forward requests for unknown modules to npmjs.org (there's a handful of ways and registry implementations that do this)
With the proposed change... you still have to do this, right? The "@" syntax may be mapped in your local .npmrc to say "use registry npm.mycompany.com for packages like @mycompany/someprivatething", but that doesn't help for internal forks of public modules.
It's possible that the above use case is out of scope for scoped modules, but do I miss something?
Scoped modules help people to have modules that are entirely private, and not risk namespace collision, and configure a private registry (if they want) to host packages with a specific scope?
But scoped modules don't deal with temporary forks of public modules, that still requires a private FORWARDING registry to be set up? (or github refs)
I'm interested in this as well and I like the @org/foo syntax. It looks clean and it's easy to grok.
My only issue with the @ & the / would be to test it on the CLI, especially on windows, and from shell scripts to ensure that the character is escaped properly and it doesn't break if you are installing CLI tools via: npm -g install @org/foo@1.2.2.
Looking at that last command makes it look a little funny now that I typed it.
Just to put a little perspective on it from my side, we've had our internal registry for a few years now. We have over 700 internal modules in our registry and we only have 4 that have name collisions. And that's simply because those modules were created before the open source modules were. We also don't allow developers to publish directly to the registry, our CI/CD system handles that.
I'm not saying that we wouldn't use it, I like it, I'm just saying that it's not been that big of an issue for us. In fact, I've actually used the single naming in the past to publish an internal module that fixes a broken or mis-configured external module so my devs could continue to work.
We've also seen the issues of working off of patched forks while we wait for the developer to accept our PR and publish a new version. Doing that with @org/ prefixes will be a PITA.
Not to mention that while typing this, I almost included several people in this thread due to the @ sign, so there's that too :)
@indexzero: You'll have to forgive me, I'm answering out of order here:
Where does it live?
If you don't have any custom mappings locally, it will live as a public package on the public registry.
Does it only exist privately if I run my own private registry? Or is there a place where it is hosted?
The behavior of the public npm registry w/r/t to privacy/package visibility isn't fully defined yet (outside of support for publicly visible @org packages).
That means that to make a package private, a client may add a mapping for @org to a private registry in their .npmrc. There may be support on whatever default registry (npmjs, nodejitsu) eventually for setting visibility of an @org, but that's not part of this issue (as I understand it!)
And more importantly: where does it live it I decide to start making it private?
Anything you published to public registry already will remain public. Anything published after the client has added the mapping between @org and a private registry url will be private (or at least, only available at that other registry).
The sequence will not be to work on @org/foo in private, and then publish it as foo.
The sequence will be to work on @org/foo as a private module, and then make @org/foo public.
If we're assuming that the majority of open source plugins are going to be in the global space—which so far we have—this seems very wrong to me. We're effectively saying that @org/foo, even if open source and public, shall have no value to other open source modules.
Echoing @davglass, while we don't have 700 modules—we're quickly approaching 100—we haven't had any issues with naming conflicts between our private modules and public ones, excepting in the case where we explicitly want to fix modules somewhere deep in our tree while we wait on the maintainer to fix. This is a very real non-temporary issue for us.
If we're assuming that the majority of open source plugins are going to be in the global space—which so far we have—this seems very wrong to me. We're effectively saying that @org/foo, even if open source and public, shall have no value to other open source modules.
That's a good point. Either global modules should be allowed to depend on @org/foo, or making new global modules should be deprecated going forward so that new modules can freely depend on new open-source namespaced modules.
I'm +1 on deprecating global namespaces and requiring that all packages live under a top level name space. That top level name space could be used as a vendor/org, username, etc. Generally this is a requirement or best practice in other languages that have used name spacing. Strictly speaking the NPM implementation doesn't have to care whether it's a user or an organization.
This also removes the above sub-module reference problem because sub-module references would become, for example, require('kriskowal/q/queue'); Any reference to require('q/queue'); would reference the q organization/user's queue package.
Users who wish to maintain authorship in the name space can continue to do so. Those who wish authorship to remain in the copyright or contributors file, instead conveying ownership/publisher in the namespace, can do so as well.
Lastly, it makes policy writing dead simple for anyone writing a NPM server, proxy, cache, client, etc. who wants to clearly dictate the mobility of an entire organization's namespace.
Cheers.
Either global modules should be allowed to depend on @org/foo, or making new global modules should be deprecated going forward so that new modules can freely depend on new open-source namespaced modules.
I think this is a problem that will solve itself over time. It allows the current module system to remain forward-compatible, and creates a cowpath by which new modules will gradually adopt the new system. Assuming the registries and clients all enforce the rules, it requires very little else be done by developers, which makes it feel like a pretty natural evolution of the system.
I'm curious about the implications this will have on the eventuality of a module being abandoned by its original author and subsequently adopted by another. Maybe this has already been discussed, but I'm too lazy to dig through the bikeshedding.
What will the process of transferring authorship look like in the future of namespaced modules?
@nexxy I don't think much has changed (yet?). Ownership transfer of modules without org prefixes will continue the way it always has. If somebody wants to take over development of a prefixed module, they could just publish it under their organization. There's still only the one namespace, it's just segmented differently.
That makes sense for 'non-prefixed' modules.
I'm more curious about what happens when 'org authorship' changes. Does @thisOneFancyOrg/someKindaModule become @someOtherFancyOrg/someKindaModule upon authorship transfer? Will there be some sort of 'symlink'? How will org-module consumers know when this happens?
Maybe this kind of thing won't happen that often, but it sounds like it could be confusing and potentially cause sudden breakage.
$ npm install (org)name
bash: syntax error near unexpected token `('
No dice, sorry.
@thomblake
Either global modules should be allowed to depend on @org/foo, or making new global modules should be deprecated going forward so that new modules can freely depend on new open-source namespaced modules.
Yeah, we can re-evaluate that a while down the road, say in a few years or so. I think we may end up opening the dependency graph to go in both directions, and/or perhaps closing the door on new global modules. But neither thing will happen soon, and by the time it does, we'll have lots more information at our disposal.
Yes, this does not address the "override xyz module for our bugfix" issue. However, it also doesn't make it any harder to do that, if you have a private registry. It will prevent those rare collision cases from happening, even if you use very generic names like logrotate or markdown or querystring, without having to publish via a CI.
My hope is that we finally put an end once and for all to the "all the good names are taken!" complaints :)
Also, keep in mind, if you're running a private registry already, you might not need this feature, or it might basically change nothing for your use case. But for a lot of other users who want namespaced modules, and especially those who may want private modules without running an entirely separate registry, or want to run a private registry for a few modules but not for all of them, we hope this will be a useful tool.
Good questions! We hadn't thought about that!
Does @thisOneFancyOrg/someKindaModule become @someOtherFancyOrg/someKindaModule upon authorship transfer?
Yes.
Will there be some sort of 'symlink'?
301 Redirect, yes.
How will org-module consumers know when this happens?
Maybe the client can print out a warning when this happens?
Does @thisOneFancyOrg/someKindaModule become @someOtherFancyOrg/someKindaModule upon authorship transfer?
- Person at @thisOneFancyOrg expresses an intent to transfer module to @someOtherFancyOrg.
- Admins of @someOtherFancyOrg receive a message that the transfer has been requested.
- @someOtherFancyOrg admins signal their intent to accept the transfer.
- Hej presto! The module belongs to @someOtherFancyOrg!
Will there be some sort of 'symlink'?
This should work at least as smoothly as GitHub does it today. Requests to the module at its original location are redirected to the module at its new location.
How will org-module consumers know when this happens?
¯＼(º_o)/¯ iono! Probably some mixture of the npm CLI silently rewriting to package.json when --save is used and printing out warnings on install.
I'm also not sure I fully understand the benefits of the additional complexity.
If orgs will be maintaining their own registry anyway (I assume?); what does this schema provide over the approach of simply mapping 'overrides' to global modules in the rare instance that it's necessary? Private modules are already available via private org registry without any namespace.
The only missing component I can identify is the ability to pull from multiple registries. That seems like something adequately solved by proxying requests for unavailable modules to a 'global' registry (one might even cache versions; faster on subsequent installs, and saving bandwidth).
Maybe I just don't understand the nature of enterprise-y needs well enough (strong possibility), but the more I read about the proposed solutions; the more I feel like there is a simpler way...
Unless this is a lead-up to the ACL/roles stuff mentioned on the original post; in which case I understand the need for these changes.
@othiym23 @isaacs thanks. these things make sense within the context of the proposed changes.
I also notice you mention orgs that are already running their own registry, which makes me wonder if I assumed the right things.
Is this to say that orgs which want their own private modules, but don't already host their own registry will be able to do so via an NPM-provided facility? That would make sense, I suppose! I guess this is where the mapping of @org -> org.something.like.whatever.npm-registry.com comes in?
Is this to say that orgs which want their own private modules, but don't already host their own registry will be able to do so via an NPM-provided facility?
Yes. The first full implementation of this will be an on-premise registry appliance-type thing.
I guess this is where the mapping of @someorg -> org.something.like.whatever.npm-registry.com comes in?
Yes. The mapping from @someorg → org.something.registry.whatever.org will be set in client configuration (perhaps in a site-wide way for larger installations).
Yes. The first full implementation of this will be an on-premise registry appliance-type thing.
very fancy indeed. I support this endeavor; particularly if you make them look like little toasters.
carry on then!
+1 to toasters
On May 9, 2014 5:37 PM, "Emily Rose" notifications@github.com wrote:
Yes. The first full implementation of this will be an on-premise registry
appliance-type thing.very fancy indeed. I support this endeavor; particularly if you make them
look like little toasters.carry on then!
—
Reply to this email directly or view it on GitHubhttps://github.com//issues/5239#issuecomment-42726525
.
If orgs will be maintaining their own registry anyway (I assume?)
You assume incorrectly! Many orgs will not be maintaining their own registry.
We spent a few months talking to a lot of companies that use Node about how they're using Node. There is a pretty sharp split between "We will never host our code in your registry" and "Please don't make us spin up another thing, just shut up and take our money". (Similar to the companies that use GH Enterprise vs those that use GH orgs and private repos.)
Every company we talked to both (a) fell into one of those buckets, and (b) didn't think it was reasonable to use the other. And the split was about 50/50. I suspect that smaller shops and individuals will be even more strongly on the "I will never run a private registry" side of the divide.
The only missing component I can identify is the ability to pull from multiple registries. That seems like something adequately solved by proxying requests for unavailable modules to a 'global' registry (one might even cache versions; faster on subsequent installs, and saving bandwidth).
You still have a single registry in that case, just one that gets some of its data from registry.npmjs.org.
Unless you mean that the client sends 404-ing requests to the public registry, but that's an information disclosure risk, and also much more complexity in practice than the "additional complexity" you're referring to.
This is indeed a lead-up to the ACL/roles stuff, as well as a way for organizations and users to have simple names that are namespaced to their owner, which has been a very frequent request over the years.
I keep getting scooped by @othiym23 who's answering questions very well. I will stop multitasking and focus on the other things I should be doing :)
@isaacs all of that sounds quite reasonable. thanks for taking the time to explain!
I will let you get back to the toasters.
As far as I can tell "global cannot depend on scoped" constraint is purely due to backwards compatibility concerns.
If no path separator is used in the scoped module name then BC can be preserved while meeting the above constraints.
I have not yet done any research into these options, but here are some ideas:
- org:module
- org=module
- org.module
- org@module
- org~module
The theme being that the special character becomes the separator rather than the prefix.
I think the above will not break in shells or URLs, but I'm unsure about clashes with existing naming.
@glenjamin The backwards compatibility concerns have more to do with the mechanism that maps @someNormalOrg to registries. Whatever characters we use to separate organizations from the rest of the package specifier, older versions of the client aren't going to recognize prefixes.
I was mostly thinking about the case where a public module is scoped.
Consider if a useful module gets published to @express/amazing-middleware
Now, in order for my unscoped module to depend on this I either need to scope my module, or convince the author to move/copy to the global namespace.
It seems reasonable to assume that a private registry will require a newer client, so if you start doing private things you can just tell your enployees to upgrade their clients when they're configuring them with the knowledge of scoped registries.
If public scoped modules don't have a directory separator, would older clients "just work"?
@glenjamin The restriction on which way the dependencies has to do with resolution of packages to registries, not with the capabilities of the clients or the server. Since part of the intention behind scoped packages is to support private packages, we don't want to create an interoperability footgun where people using old clients end up grabbing dependencies from the wrong registry. As @isaacs said, we may reassess this restriction in the future.
My understanding from @isaacs earlier in the thread was that an org publishing conflicting modules publically and privately with the same name and number was considered "doing it wrong", and wouldn't be explicitly stopped by clients.
It's been stated that clients will have the knowledge of what @org maps to what registry - so even with an up-to-date client I can end up in the "footgun" you describe just by not having the same client config as other people.
My point is that restricting global -> scoped dependencies, even if only to begin with, seems to add complexity and inconsistent behaviour for not much (any?) gain.
My understanding from @isaacs earlier in the thread was that an org publishing conflicting modules publically and privately with the same name and number was considered "doing it wrong", and wouldn't be explicitly stopped by clients.
This is how it's supposed to work, yeah. How does that pose a problem?
My point is that restricting global -> scoped dependencies, even if only to begin with, seems to add complexity and inconsistent behaviour for not much (any?) gain.
This is something designed to make things simpler for npm users in general by making them a little more complicated for scoped module publishers. We're optimizing for the common case, and making a bet that over time, people will move to using scoped modules in general. How is this inconsistent?
Using a separator (other than /, that is) rather than a prefix is an option we did talk about, but perhaps could re-open and look at a bit more closely.
org:module
Breaks on windows file systems because : is magical there.
org=module
Possible.
org.module
Already a valid name, so can collide with non-scoped packages.
org@module
Collides with name@tag, so not disambiguable.
org~module
Same problem as ..
An upside of using a forward slash is that naturally groups things by org on the filesystem as well, which is nice for operational reasons. Not insurmountable either way, of course.
Since = is not already allowed, it could work as a separator just as well as the @org/pkg format.
The question then would be whether yahoo=yui is as clear as @yahoo/yui. My brain wants to say that one is "the entity named yahoo, and underneath that, the module named yui", and read the other as "yahoo equals yui". But maybe that's just habituation, and we'd get used to it? Worth thinking about some more, I suppose. What do other people think?
Gah, I just wrote a nice long reply then fat fingered the button and lost it!
I'll have a bit more of a think and another read-through to digest the thread, but at the moment I'm still struggling to understand what's gained by restricting this case.
This is how it's supposed to work, yeah. How does that pose a problem?
I don't think this is a problem, but I'm not seeing how the footgun scenario you describe is caused unless this is the case.
The inconsistency is that global and scoped modules are only allowed to interact in 3 out of 4 possible ways.
My concern is that this actually makes things harder for global module publishers, as there would be this public list of modules that some people can use, but they cannot unless they rename their module.
I hope I'm interpreting / explaining this correctly, I did not intend to get into an argument :)
@glenjamin Part of the goal here is to nudge people towards using scoped modules, but also prevent accidentally screwing up the current ecosystem of global OSS modules. None of this is set in stone, but if we rush in and open the floodgates right away, it can cause problems that are difficult to back away from.
Once org-capable npm clients are in common usage, it'll probably be fine to let the dependency flow go in both directions. I just want to be cautious about how we roll it out.
That was something I had in the draft I lost - an acknowledgement that taking it away in the future is much harder than starting without and adding it later.
A slightly off-the-wall suggestion: would making module equivalent to @global/module or similar ease the migration path for existing unscoped modules?
I had a quick look at https://github.com/npm/npm-registry-couchapp/blob/master/registry/modules.js
Assuming this is the correct validator to look at it appears = is currently allowed, but ~ is not. yahoo~yui appears marginally less weird than yahoo=yui imo.
Assuming this approach is explored further, is the constraint "characters not currently allowed in package names" or "characters not currently used in package names"?
@glenjamin There's also the URL encoding and the CLI usage to think about as well. Some characters can't be used in both or they break. So it's "characters not currently allowed in package names, characters that are allowed on the filesystem/cli on all OSes without escaping and characters that are allowed in URL's without escaping and it needs to be easily understood that it's a grouping"
It's a hard choice to make since it needs to fit in all of those places :)
A slightly off-the-wall suggestion: would making module equivalent to @global/module or similar ease the migration path for existing unscoped modules?
👍 having request be an alias for @global/request would make refactoring of existing un-scoped modules easier.
It would also hopefully make the implementation easier, basically deprecate unscoped modules and add a special case that treats unscoped modules as @global/{whatever}.
I also just squatted https://www.npmjs.org/~global for the purposes of this feature.
The fallback mechanism that allows for global packages to depend on scoped packages is actually quite simple:
We're not going to send 404-ing requests to the main registry. Doing that would be an information disclosure vulnerability.
I believe you misread my flow @izs. I've restated it as the diagram below:
The 404s would only go to the registry explictly set by the user for a given @org. So information is only disclosured to a registries explicitly specified by the user. In other words: they're telling us to disclose said 404 "tombstone"-like information so it cannot be exploited.
Once org-capable npm clients are in common usage, it'll probably be fine to let the dependency flow go in both directions. I just want to be cautious about how we roll it out.
That is a fine way to do this @izs. I would be very happy if this was the way this was implemented and then the timeline for when to expect these features could be established.
@indexzero Unless I'm misunderstanding your graph in the same way, I did understand the workflow you described. Item (2) under If REGISTRY config for @org is an inappropriate information disclosure, that's exactly what I was referring to. The client has asked to only send those requests to a specific registry other than the main registry. We're not going to fall back from one registry to another in the case of a 404. That is fraught with both technical and moral hazard.
@glenjamin Aha, the server doesn't allow tildes but the client does. Qv: https://gist.github.com/isaacs/2de230ae78529313f553
@glenjamin There's also the URL encoding and the CLI usage to think about as well. Some characters can't be used in both or they break. So it's "characters not currently allowed in package names, characters that are allowed on the filesystem/cli on all OSes without escaping and characters that are allowed in URL's without escaping and it needs to be easily understood that it's a grouping"
That sounds like an accurate list of the constraints.
It's a hard choice to make since it needs to fit in all of those places :)
On the contrary, it removes almost every possibility, making the choice mostly a simple process of elimination.
@glenjamin btw, re my last comment, the reason why "works on the server, not on the client" is relevant is that it's much easier to fix the server-side logic than the client, so that's less of an issue.
So, ~ is worth exploring, for sure!
The client has asked to only send those requests to a specific registry other than the main registry. We're not going to fall back from one registry to another in the case of a 404. That is fraught with both technical and moral hazard.
Could you elaborate on this? Fallback can be very graceful. For example, this is exactly how smart-private-npm handles publishes of new packages:
// - if it does not exist we proxy to the private registry
// - if it does exist then we proxy to the public registry
We've been using this workflow in production for many months now with no technical or moral concerns from our own petri dish of experimentation. The core reason why this works is that the user has two distinct expectations for an @org scoped module:
- It is private and therefore should be in the
--registryconfig for the@org, which has an expectation to not be the main public registry. - It is public and therefore would be in the main public registry at the default
--registryconfig or the global default--registryatnpmjs.org
Do you disagree with these expectations? I can see scenarios of "there be dragons" that are not built on these assumptions, but I have not heard of other common assumptions from anyone I've spoken with about this.
The thing about "open development" is that you get to see stuff before all the questions are answered. One might say that's the only thing about open development, in fact! Expect some partly-baked experimentation.
In the interest of keeping this conversation useful for newcomers, and not them scaring away with an even more impenetrable wall of text, any further messages that reintroduce already-answered questions may be deleted or edited at the discretion of the npm project maintainers. Thanks.
We're discussing a really big change to how npm works. It's going to be and should be a through discussion. In the interest of making this more approachable, I am going to break up my comment sub-sections into individual comments. Alternatively, if you'd like to reach out to me privately, you know where to find me.
I'm also going to take a page from @domenic's book and I tried to summarize the discussion so far here.
While in theory I can see the information disclosure issue, in practice the request would only hit the default registry when the package doesn't exist privately - so you're only potentially revealing the names of packages that dont exist privately.
Without this a publically scoped module in the same scope as some private modules would have to be mirrored internally, or the org would need to name their public and private scopes differently.
Not that these two options are terrible, but I can see the argument for fallbacks. Perhaps opt-in from client config but off by default?
Just a ponderance, but this seems to have skipped some things like behavior of people nesting directories/scopes with the same syntax as organizations; for example, @org/@ou/module, seems to be perfectly valid on the filesystem (perhaps not the registry) but could cause some confusion. This is probably minor and "doing it wrong", however it does seem to have some truth in scopes sometimes having more internal scopes.
Say I do npm install @NSA/my-secret-project-name. Then due to a server
misconfiguration the private registry sends a 404. Now should the client
send a request for @NSA/my-secret-project-name to the public registry, or
should it just say not found? I'm pretty sure sending your secrets to
other people's servers is the thing not to do.
On Sun, May 11, 2014 at 2:32 AM, Glen Mailer notifications@github.comwrote:
While in theory I can see the information disclosure issue, in practice
the request would only hit the default registry when the package doesn't
exist privately - so you're only potentially revealing the names of
packages that dont exist privately.Without this a publically scoped module in the same scope as some private
modules would have to be mirrored internally, or the org would need to name
their public and private scopes differently.Not that these two options are terrible, but I can see the argument for
fallbacks. Perhaps opt-in from client config but off by default?—
Reply to this email directly or view it on GitHubhttps://github.com//issues/5239#issuecomment-42766120
.
@thomblake Here's the thing that I don't get about this argument. Yes, acknowledging the existence of package @org/foo isn't ideal, but if we are allowing for user misconfiguration errors then accidental information disclosure can happen across the board, whether or not global modules are allowed to depend on scoped modules.
e.g.
- I have some project that depends on
@NSA/my-secret-project-name - My npm client is misconfigured, and lacking information about the
@NSAorg. - I still make the request to the public registry disclosing the existence of
@NSA/my-secret-project-name
So since the information disclosure can happen on any npm client that is misconfigured regardless, what exactly is the problem with allowing global modules to depend on scoped modules? Is there a scenario that allows for information disclosure vulnerability that isn't "the user misconfigured their client"?
Is there a scenario that allows for information disclosure vulnerability that isn't "the user misconfigured their client"?
Yes. "The user's private registry vendor wrote a buggy server". The npm client should be resilient to that general class of mistake, if we can actually anticipate it ahead of time.
Also, devs at this NSA company will be able to put a npmrc file in the root of all their projects that defines the internal registry location, or even pre-configure their devs' computers with this config pre-installed.
"Misconfigured client" is a very solvable problem. "Typoed the package name" is not. If the client directs 404s to the public registry, then that's terrible.
If you're currently doing this in your product, I'd suggest that's a pretty significant bug you ought to disclose to your customers.
From within the NSA, would I set the package name to "@nsa/secret-project" and global npm module dependencies would be "mochify"?
Or would I set the package name to "secret-project", have my default registry set to the internal one, and name my global dependencies "@global/mochify"?
I would find it slightly confusing if I name my project "secret-project", npm publish it under that name, but have to add a dependency named "@nsa/secret-project". That is likely to go wrong and lead to the described information disclosure problem.
Also, devs at this NSA company will be able to put a
npmrcfile in the root of all their projects that defines the internal registry location, or even pre-configure their devs' computers with this config pre-installed.
+1 to this and deprecating publishConfig. Solving config problem(s) using a dotfile makes much more sense.
"Misconfigured client" is a very solvable problem. "Typoed the package name" is not. If the client directs 404s to the public registry, then that's terrible.
To be clear there is no information disclosure vulnerability in a properly configured npm client using the mechanism suggested. In this perfect world the private registry config for @org is always set properly and the requests to the public registry on 404 are intended, not an unintended disclosure of any kind.
It's tradeoffs. Perhaps hundreds of bad 404s would be made. Hundreds out of billions of total requests. Yes, the information is there, but it's a needle in a haystack and the information itself is not that sensitive. It is not code: it is knowing that @org has a module named foo. Not even where the real private registry that holds @org/foo is. This of course presumes that the npm client drops the Authorization headers on your GET requests to the public registry (which is a must here regardless imho).
Beyond that: this information disclosure will happen one way or another. Even the best of the best Node.js developers make fat fingered mistakes pretty frequently. publishConfig helps, but makes projects rigid when you switch registries.
The upside of this calculated risk is that scoped modules and existing global modules can live side-by-side peacefully depending on one-another and generally not confusing people. It's a risk that is highly worth it. What are the thoughts on syntax that explicitly says a module is private or public? e.g. git+ssh:// already tells the npm client with some certainty that "hey, this thing is secured." What about either of these:
{
"dependencies": {
"@NSA/prism": "private ^1.2.3",
"@NSA/prism": "p ^1.2.3"
}
} Making an explicit distinction between "scoped public modules" and "scoped private modules" may be just the thing here.
In this perfect world the private registry config for @org is always set properly and the requests to the public registry on 404 are intended, not an unintended disclosure of any kind.
I'm bound by NDA to not say too much. But, just to be 100% clear on this: I have personally spoken with technical decision makers at real companies who have told me that they would be uncomfortable if the client "fell back" to a public registry for 404s. If you say "@org/..." modules should be hosted in your private registry, and a user types @org/foo instead of @org/fop, then it should simply fail, rather than tell npm, Inc. that @org exists, does not have a module named foo, and probably does have a module named something close to foo.
So, we'd have to provide a clear way to opt-out, as well as disclose it. And frankly, building something that adds complexity and isn't desired by most users is really not something I'm too keen on anyway.
Your suggestion has been heard, the problems elaborated by me and others. If you don't have new information to provide, let's close that thread and move on.
From within the NSA, would I set the package name to "@nsa/secret-project" and global npm module dependencies would be "mochify"?
Yes, scoped modules can depend on global modules.
Or would I set the package name to "secret-project", have my default registry set to the internal one, and name my global dependencies "@global/mochify"?
No, I think introducing a "magic" namespace called "@global" is unnecessary complexity.
I would find it slightly confusing if I name my project "secret-project", npm publish it under that name, but have to add a dependency named "@nsa/secret-project". That is likely to go wrong and lead to the described information disclosure problem.
In the proposed scheme, a non-scoped module like "secret-project" woudln't be able to depend on a scoped dependency like "@nsa/secret-project"
@isaacs I sympathize with the desire to not query npm, inc's registries with "@nsa/privacy-violators" when it isn't found on the private reg configured to be used for "@nsa", we hear similar things from clients. But if the desire is to not leak info to Npm.inc on accidental mis-config/mis-type, I don't think you've achieved that, you only plugged one hole (misnamed modules).
If I forget to configure "@nsa" to go to nsa's registry (unlikely on a dev machine, more likely on vms), and npm install an app that uses "@nsa" scoped modules, it will send every single dependency's name to Npm,inc. Oops.
Obvious way to fix this is to not send any queries to a registry unless it is configured to serve that scope.
This would conflict with making it easy to publish/subscribe scoped modules to npmjs.org.
Seems like a requirements conflict to me.
@sam-github It's plugging the hole we CAN plug, which incidentally also the much more difficult hole for the client to fix on their end. Human fingers will do the wrong things repeatedly. But dev environments and package folders can be deterministically configured with a high degree of reliability.
And for the truly paranoid, they would be able to use npm://nsa-registry.local/@nsa/fop to refer to the package, which is what @nsa/fop would desugar to anyhow. So, as long as the client doesn't try to be overly clever in that case, there's zero risk.
Thanks for the explanation @isaacs.
From my perspective from within an "enterprise" environment (and a potential npm customer), I would prefer it to be the other way around. Let me explain in more detail.
The @org feature, as discussed so far, is used for two things:
- namespacing
- registry routing
I think it's going to cause quite some confusion if @namespace and @repo are used side by side in the dependencies. One pointing to a namespace in the central repo, the other one being private.
If we take a step back and think about only doing the routing, we could have a much simpler and clearer solution:
- "my-project" refers to whatever my default repository is
- "@repo/my-project" refers to whatever "repo" is configured to be
Looking at our projects, we have 90% private modules, and they must stay within the company network. It would feel natural to only mark the "external" dependencies. If there is a need for another department within the company to have their own repository, that would work as well.
We already want to have two synchronized repositories for performance, deployed in different locations. They would have different URLs, but the contents are shared. We could solve that by having different default repo configs per region and make @npm point to the central.
Does that make any sense? Or am I getting it all wrong?
This seems like an interesting slant on the problem, I think it's worth noting again at this point that we have two viable syntax choices, perhaps using one each for these two features might be a good approach?
Unsure if that makes things more complicated, or actually does manage to decomplect the concepts of private registries and namespaces.
I'm bound by NDA to not say too much. But, just to be 100% clear on this: I have personally spoken with technical decision makers at real companies who have told me that they would be uncomfortable if the client "fell back" to a public registry for 404s.
So, let me see if I follow you. Companies that willfully disclosed to you and npm Inc. that they are using node and npm are concerned about disclosing this information to you again in the future? I hope you can see how two dimes and a nickel aren't making a quarter on this here.
So, we'd have to provide a clear way to opt-out, as well as disclose it. And frankly, building something that adds complexity and isn't desired by most users is really not something I'm too keen on anyway.
Philosophically "the needs of the many outweigh the needs of the few" is easy to agree on, but are we sure that this is the need of the many? There are only ~20 people on this comment thread, many of whom (like myself) do desire this. How do we know "global can depend on scoped modules" is not desired by most users? I'm happy to help get the word out and try to distill a public opinion on this. This survey only got 13 responses, but that was only a single tweet looking for answers.
Search for omnipresence aside, making this opt-in could meet mutual goals here. Module consumers who wish to use global modules that depend on scoped modules would be forced to enable it, so it would be up to the module authors (as it always has been) to decide whether or not this is a worthwhile feature to them.
Rather than try to overload package names, has there been any consideration to re-archicture of package.json dependencies and dev/optional/bundled/peer/whatsNext?-Dependencies??
as a off-the-cuff example of changing the shape ...
{
"repos": {
"myCo": "http://npmjs.myco.wtf",
},
"deps": {
"runtime": {
"foo": "0.1.x",
"bar": "myCo:0.2.x"
},
"dev": {
"blortz": "0.3.x",
"pinky": "myCo:0.4.x"
}
}
}Seems like it would be straight-forward-ish to prototype this with an install script in the package.json
The @org feature, as discussed so far, is used for two things:
namespacing
registry routingI think it's going to cause quite some confusion if @namespace and @repo are used side by side in the dependencies. One pointing to a namespace in the central repo, the other one being private.
I disagree. @org/pkg is for namespacing only. A property of a namespace is that you can specify which registry it lives in, but multiple namespaces can live in a single registry. The top-level --registry is then effectively specifying just the home for the global modules, and namespaces that do not have some other home specified.
Remember, @org/pkg doesn't expand to https://registry.my-org.com/pkg, but rather https://registry.my-org.com/@org/pkg. So, you can still have multiple different organizational namespaces within a single internal registry.
Looking at our projects, we have 90% private modules, and they must stay within the company network. It would feel natural to only mark the "external" dependencies.
What you may find is that non-namespaced modules start to stand out after a while. Already, I have a ton of npm-this and npm-that. I'm starting to come around to @visionmedia's pov that it'd be nice to just have @npm/this and @npm/that and then be able to manage and look at them more easily as a single group of things.
@glenjamin Using one syntax for namespacing, and another for registry location, does indeed increase complexity. There will already be a syntax for specifying the registry location, using npm://registry.my-org.com/@foo/bar, if you want to use that.
So, let me see if I follow you. Companies that willfully disclosed to you and npm Inc. that they are using node and npm are concerned about disclosing this information to you again in the future?
You are not following me.
Companies that willfully disclosed to me, consciously and with careful foresight and executive involvement, while I was under NDA that they are using Node and npm. They are concerned about disclosing additional information about the details of their private use of Node, via automated and uncontrolled mechanisms, without executive oversight or legal protection.
That subject is closed. Thanks for your suggestion. It's been explored, the problems elaborated, and it has been rejected on the basis of those problems.
If you want to explore doing that kind of behavior, you can build your own registry implementation which does exactly what you are describing (either via a 302 response, or a proxy), without changing the client at all. In fact, I believe you already have. So, I'm not sure what there is to even discuss. You asked a question, and got an answer.
Regarding the second suggestion, that we repurpose foo/bar instead of @foo/bar for the use of scoped modules:
The issue isn't just backwards compatibility, either. We benefit by making sure that the new thing works and fails crisply when it can't work, rather than doing a different thing, and encountering hazards when users of new-npm (where foo/bar is a scoped module) try to consume package.json files created by users of old-npm (where foo/bar is a github repo). That's just a recipe for disaster.
We may break backwards compatibility if there are clear benefits, but in this case, the benefits are much less than the costs. In fact, I don't even really see what the benefits are, except for typing the @ sign. So, no, not going to happen.
At this point, continuing to restate settled issues is approaching trolling. Your points have been made. Please make new points, or just let it rest.
Rather than try to overload package names, has there been any consideration to re-archicture of package.json dependencies and dev/optional/bundled/peer/whatsNext?-Dependencies??
We did discuss that internally, but it hasn't been brought up yet in this thread.
There are some advantages to adding to the package name namespace rather than expanding package.json shape.
- We don't have to teach people as much new stuff. The existing pattern of "you put
Xin package.json dependencies, you dorequire(X), andnpm install Xinstalls it" continues to hold. That's a pattern that has proven successful. - It's a lot less surgery in npm, the registry data validation scripts, etc.
That's not to say that there isn't a use case for re-thinking some of the things in the package.json data structures, but moving that stuff around is hazardous and in this case, not the simplest way to achieve the desired goal.
@pmuellr Another advantage of putting this data in the cli and require() statements explicitly is that it's hinted at a much higher "level" in a development process. Rather than edit package.json to add a second repo, and then do npm install foo, and know that foo is actually in the org repo, and manage two (potentially conflicting) require("foo") things in my JavaScript, you get all that info hinted from the very start, and throughout the process. So, you do npm install @org/foo and require("@org/foo"), and conflicts are significantly less of a problem.
@isaacs Thanks for taking the time to explain. I think it's going to work for me as suggested.
I tried to follow this discussion and try to make my point here now, apologies for bad language.
I don't feel quite comfortable with this whole thing. I don't like the fact that @SomeCompany should be a namespace. if i want to namespace some packages i can do it, today like myns-mypackage. namespacing is all about prefixing of names. this is a non-issue imho.
private packages is an issue and here i would go with the prefixes notation, but for repository resolution only. so if i put @somens/somepackage as a dependancy, then i either define that @somens points to a specific repository, or i let npm resolve this name to an url from a repository registry. this would imply. This has some benefits:
- If you want to keep your package private, you can either host your own repo (fits @isaacs most paranoid customers), or buy a hosted one as i.e. nodejitsu is offering (fits the shut-up fraction). then just add a repository to your local client and go on.
- if you want to publish you private package later, you can just do it on main npm.
- every repository can provide public access to a subset of packages. if they do it so, they can register themselves as public repos in main npm, so npm can resolve
@namespaceto repos - no need for 404 disclosure in this case: 1. look at the private repo, 2. look if there is a public repo for this namespace. 3. if there is one, load it from there. if not, spit out an error to the client, because in this case something isn't configured right
i hope i made my point clear here. :)