standard/standard-packages

Switch to libnpm instead of curling the couchdb endpoint

feross opened this issue · 5 comments

npm removed their couchdb endpoint, so now we need to list all package names from this endpoint (https://skimdb.npmjs.com/registry/_all_docs) and then use libnpm (https://www.npmjs.com/package/libnpm) to download each package and look for standard dependencies.

What do you think about using package-stream? I did a quick test run and it happily found hundreds of packages listing standard in their devDependencies. I’d be glad to open a pull request.

Edit: On second thought, I see that the list also includes the number of dependents which might be difficult to retrieve with this approach.

@sonicdoe Nice find. I should have known about package-stream!

Could we retrieve the number of dependents by running through the list of all packages twice?

First time through, we make a list of every package that depends on standard. Call this standardPackages.

Second time through, for each package, we check if it depends on any of the packages in standardPackages. If so, increment a counter for that specific package.

Might be slow, but it would at least work and we don't run this script very often. I usually run it anytime a major version is released to get an updated count of users and to see who the most popular users are.

If you want to open a pull request and take a stab at this. I'm planning to release standard v14 this weekend, so it would be nice to have this by then if you have the time :) But no pressure. We can always get to it later.

I’ve decided to go with a slightly different approach in that I replicate the npm registry to a local CouchDB database, see #31. Let me know what you think.

@sonicdoe This is excellent! Thank you!