Add support for npm5

Question

Add support for npm5

JamieMason opened this issue 7 years ago · 86 comments

npm install -g npm5 has a different cache structure so shrinkpack will need updating to support that, while being backwards compatible for older versions.

Answer 1 · 2017-05-07T08:59:03.000Z

So, there's two ways you can go about this: The way I think I most recommend is to run npm pack pkg@1.2.3 pkg2@3.4.5 other-pkg@1 ..., according to the versions in npm-shrinkwrap.json or package-lock.json. This way is compatible with just about any version of npm I know of, and does not even require a warm cache to work: it'll use the cache if needed, but will otherwise make sure whatever you need to fetch gets fetched before it's copied to the cwd.

The other way, which will be only compatible with npm@>=5, is to use cacache in conjunction with the content addresses in package-lock.json or npm-shrinkwrap.json. That is, cacache.get.stream.byDigest('~/.npm/_cacache', locked.dependencies[pkg].integrity).pipe(fs.createWriteStream(locked.dependencies[pkg].name + '.tgz'). Obvs with whatever other stuff you wanna add to this. Note that doing it this way is not all that different from going the npm pack route, except this version allows you to use/verify integrity hashes when available.

lmk if you have any questions about the cache format or the tools around it. shrinkpack is real cool. :)

Answer 2 · 2017-05-07T09:55:57.000Z

Thanks @zkat, I'll most likely go the npm pack route 👍

Answer 3 · 2017-05-29T09:37:18.000Z

In the meantime, it might be helpful to pass {stdio: ['pipe', 'pipe', 'inherit']} when you spawn npm so stderr doesn't get swallowed. Anyway, I appreciate the effort on this. 👍

Answer 4 · 2017-05-29T19:06:52.000Z

Thanks a lot for the tip @timdp

Answer 5 · 2017-06-07T07:57:21.000Z

@zkat should I consider rewriting the paths in package-lock.json in the same way I do npm-shrinkwrap.json (from URLs to their local .tar path) or would you advise against that?

Answer 6 · 2017-06-07T17:09:48.000Z

@JamieMason If you do that, it HAS to be: file:relative/path/to.tar, and you have to add the sha512sum of the .tar to integrity (NOT replace it -- it needs to have both sha512s in that string, separated by a space. See https://npm.im/ssri)

I haven't tried this myself yet, but I would hope it would work. It should. If not, that's a bug I gotta fix.

So, to summarize:

"foo": {
  "version": "1.2.3",
  "resolved": "https://registry.npmjs.org/mypkg/-/mypkg-1.2.3.tgz",
  "integrity": "sha512-deadbeef"
}

becomes

"foo": {
  "version": "1.2.3",
  "resolved": "file:package.mirror/mypkg-1.2.3.tar",
  "integrity": "sha512-deadbeef sha512-c0ffee"
}

If the integrity hash doesn't match resolved, you'll get an integrity error.

Answer 7 · 2017-07-18T17:00:17.000Z

@JamieMason just wondering, any progress on this? Thanks!

Answer 8 · 2017-07-19T11:25:59.000Z

Hey @zkat,
I've noticed a few mentions from npm peeps that there's a plan to ~implement shrinkpack in npm, is that right? If so, would there be any need to add npm v5 support to shrinkpack if it will be available without a 3rd party package anyway?

Thanks a lot,

Jamie

Answer 9 · 2017-07-19T18:43:06.000Z

@JamieMason doing that is a ways away. You're free to implement it shrinkpack-side in he meantime -- it's gone pretty far down the queue in the past couple of months, and idk when we'd even get around to it, rn.

Answer 10 · 2017-07-19T19:02:38.000Z

righto, thanks a lot @zkat

Answer 11 · 2017-07-25T13:37:43.000Z

Is this something that is in progress?

Answer 12 · 2017-07-25T14:36:13.000Z

Hey again @zkat, hope you like @ mentions.

If I wanted to keep the ↓ part of the +110 -0 ↓50 →60 ✓0 summary shown after shrinkpack has run, am I right that for npm >= 5 cacache.get.hasContent(cache, integrity) is the method I would want to use?

I'm thinking I would use that to report on whether the package was downloaded, but run npm pack to do the ✨🦄 ✨.

@pietermees @cmandlbaur It hasn't been possible to work on shrinkpack up until now, from next week I will be able to hopefully make a start.

If anyone knows of anything I can do to help make it easier for people to contribute, please let me know. I've spent a lot of time on the readme.md and contributing.md etc but there's probably all kinds of stuff that should be clear to people but really isn't.

Answer 13 · 2017-07-25T14:45:22.000Z

That's great news @JamieMason!

I'm not entirely sure what would make contributing easier. I think there's a general sense (at least on my end) that I don't fully understand all the moving parts. And seeing that even you - after all this amazing work - still seem to rely on zkat's knowledge from time to time, makes me feel like the learning curve is relatively long and steep ;)

On the one hand, having a number of smaller tasks that people can tackle to learn seems like a good idea, on the other hand people probably run into specific issues from time to time that they care about a lot (like this one) and are very happy the rest of the time. So not many people might pick up the smaller tasks.

Answer 14 · 2017-07-28T00:20:42.000Z

@JamieMason yeah. Although note that the cacache directory is inside ~/.npm/_cacache, when you point it at a cache

Answer 15 · 2017-07-28T06:22:05.000Z

Thanks again 👊

Answer 16 · 2017-08-04T14:25:57.000Z

Sorry @zkat, can I nag you again please?

1

I've noticed that sometimes entries have a semver string for version and a location for resolved
and other times a location for version and no resolved field, like so;

EDIT: package-lock.json on its own looks as expected (I'm on npm 5.3.0) but the following is seen if the user runs npm shrinkwrap --dev over it.

// familar, with version semver and resolved location
"enhanced-resolve": {
  "version": "3.4.1",
  "resolved": "https://registry.npmjs.org/enhanced-resolve/-/enhanced-resolve-3.4.1.tgz",
  "integrity": "sha1-BCHjOf1xQZs9oT0Smzl5BAIwR24=",
  "dev": true,
  "requires": {
    "graceful-fs": "https://registry.npmjs.org/graceful-fs/-/graceful-fs-4.1.11.tgz",
    "memory-fs": "0.4.1",
    "object-assign": "https://registry.npmjs.org/object-assign/-/object-assign-4.1.1.tgz",
    "tapable": "0.2.8"
  }
},
// new, with location version and no resolved
"ent": {
  "version": "https://registry.npmjs.org/ent/-/ent-2.2.0.tgz",
  "integrity": "sha1-6WQhkyWiHQX0RGai9obtbOX13R0=",
  "dev": true
},

When running npm pack, should I eg. npm pack ${node.resolved || node.version}? Can you tell me
anything more about the rules behind what's happening here?

2

When I come to update the lockfile, what should the resulting version be in these cases? eg.

// input
"ent": {
  "version": "https://registry.npmjs.org/ent/-/ent-2.2.0.tgz",
  "integrity": "sha1-6WQhkyWiHQX0RGai9obtbOX13R0=",
  "dev": true
},

Would npm be able to resolve the dependency if that became this?

// possible output 1
"ent": {
  "version": "file:node_shrinkwrap/some-file.tar",
  "integrity": "sha1-6WQhkyWiHQX0RGai9obtbOX13R0= some-new-sha-for-the-local-file",
  "dev": true
},

Or should it maybe become this?

// possible output 2
"ent": {
  "version": "https://registry.npmjs.org/ent/-/ent-2.2.0.tgz",
  "resolved": "file:node_shrinkwrap/some-file.tar",
  "integrity": "sha1-6WQhkyWiHQX0RGai9obtbOX13R0= some-new-sha-for-the-local-file",
  "dev": true
},

3

Can you point me to anything you'd recommend I should use to generate the sha512s please? also,
what should I feed it? the contents of the new tarball? and/or the location? This isn't something
I've done before.

4

EDIT: Following on from question 1, I wonder if I'll be able to maintain the file names of node_shrinkwrap/<name>-<semver>.tgz without opening the packages?

Sorry this is a huge list, thanks for all your help.

Answer 17 · 2017-08-04T23:56:19.000Z

See the spec for package-lock's version field here: https://github.com/npm/npm/blob/latest/doc/spec/package-lock.md#version-changed. Also, you should keep requires in there.
Thinking about this makes me wonder if npm5 can actually do this sort of thing right now. URL specs don't match semver specs, so npm would (I believe) immediately NOPE out and force a full reinstallation because the package-lock would not be considered to match the package.json. I think the right thing here is to keep the original version field value and change resolved, but I think npm insists on resolved being web URLs right now. I really just don't know and you'll have to experiment here and maybe help patch npm to make it support this.
https://npm.im/ssri is a full toolset for working with SRI strings, and is what npm itself uses
I don't really understand this or why it's a problem.

Answer 18 · 2017-08-05T00:22:15.000Z

Thanks a lot

Answer 19 · 2017-08-09T19:11:44.000Z

Just wanted to leave a note that I'm pleased that you've found time to work on this.

Like @pietermees:

there's a general sense (at least on my end) that I don't fully understand all the moving parts. And seeing that even you - after all this amazing work - still seem to rely on zkat's knowledge from time to time, makes me feel like the learning curve is relatively long and steep ;)

So I feel indebted to you for the work already done, and for the work you've now picked up again.

I'm cheering for you, and I deeply appreciate your time and energy on this project. 🍻 🎉

Answer 20 · 2017-08-09T19:58:34.000Z

I second that!

Answer 21 · 2017-08-10T09:29:56.000Z

That's amazing, thanks so much. Just a note to people dropping in that I'm travelling now so won't be able to pick up again until mid-late next week.

😊👍

Answer 22 · 2017-08-15T21:23:52.000Z

Some learnings today, I've created 4 branches which show different structures of npm-shrinkwrap.json and package.json.

format 1 ❌
format 2 ❌
format 3 ✅
format 4 ❌

Possible things to try next are updating the requires fields as follows...

"react": {
  "version": "file:node_shrinkwrap/react-15.6.1.tgz",
  "integrity": "sha1-uqhDTsZ4C96ZfNw4C3nNM7ljk98=",
  "requires": {
    "create-react-class": "file:node_shrinkwrap/create-react-class-15.6.0.tgz",
    "fbjs": "file:node_shrinkwrap/fbjs-0.8.14.tgz",
    "loose-envify": "file:node_shrinkwrap/loose-envify-1.3.1.tgz",
    "object-assign": "file:node_shrinkwrap/object-assign-4.1.1.tgz",
    "prop-types": "file:node_shrinkwrap/prop-types-15.5.10.tgz"
  }
}

...as that is what was produced when running the following to see how npm itself would generate a localised npm-shrinkwrap.json;

npm install --save ./node_shrinkwrap/angular-1.6.5.tgz
npm install --save ./node_shrinkwrap/asap-2.0.6.tgz
npm install --save ./node_shrinkwrap/core-js-1.2.7.tgz
npm install --save ./node_shrinkwrap/create-react-class-15.6.0.tgz
npm install --save ./node_shrinkwrap/encoding-0.1.12.tgz
npm install --save ./node_shrinkwrap/fbjs-0.8.14.tgz
npm install --save ./node_shrinkwrap/iconv-lite-0.4.18.tgz
npm install --save ./node_shrinkwrap/is-stream-1.1.0.tgz
npm install --save ./node_shrinkwrap/isomorphic-fetch-2.2.1.tgz
npm install --save ./node_shrinkwrap/js-tokens-3.0.2.tgz
npm install --save ./node_shrinkwrap/lodash-4.17.4.tgz
npm install --save ./node_shrinkwrap/lodash.assign-4.2.0.tgz
npm install --save ./node_shrinkwrap/loose-envify-1.3.1.tgz
npm install --save ./node_shrinkwrap/node-fetch-1.7.2.tgz
npm install --save ./node_shrinkwrap/object-assign-4.1.1.tgz
npm install --save ./node_shrinkwrap/promise-7.3.1.tgz
npm install --save ./node_shrinkwrap/prop-types-15.5.10.tgz
npm install --save ./node_shrinkwrap/react-15.6.1.tgz
npm install --save ./node_shrinkwrap/setimmediate-1.0.5.tgz
npm install --save ./node_shrinkwrap/ua-parser-js-0.7.14.tgz
npm install --save ./node_shrinkwrap/whatwg-fetch-2.0.3.tgz
npm shrinkwrap --dev

that produced format 3 which is the only format which will let you install while offline and with an empty npm/cacache cache. But it has the following in package.json which will not be acceptable;

  "dependencies": {
    "angular": "file:node_shrinkwrap/angular-1.6.5",
    "asap": "file:node_shrinkwrap/asap-2.0.6",
    "core-js": "file:node_shrinkwrap/core-js-1.2.7",
    "create-react-class": "file:node_shrinkwrap/create-react-class-15.6.0",
    "encoding": "file:node_shrinkwrap/encoding-0.1.12",
    "fbjs": "file:node_shrinkwrap/fbjs-0.8.14",
    "iconv-lite": "file:node_shrinkwrap/iconv-lite-0.4.18",
    "is-stream": "file:node_shrinkwrap/is-stream-1.1.0",
    "isomorphic-fetch": "file:node_shrinkwrap/isomorphic-fetch-2.2.1",
    "js-tokens": "file:node_shrinkwrap/js-tokens-3.0.2",
    "lodash": "file:node_shrinkwrap/lodash-4.17.4",
    "lodash.assign": "file:node_shrinkwrap/lodash.assign-4.2.0",
    "loose-envify": "file:node_shrinkwrap/loose-envify-1.3.1",
    "node-fetch": "file:node_shrinkwrap/node-fetch-1.7.2",
    "object-assign": "file:node_shrinkwrap/object-assign-4.1.1",
    "promise": "file:node_shrinkwrap/promise-7.3.1",
    "prop-types": "file:node_shrinkwrap/prop-types-15.5.10",
    "react": "file:node_shrinkwrap/react-15.6.1",
    "setimmediate": "file:node_shrinkwrap/setimmediate-1.0.5",
    "ua-parser-js": "file:node_shrinkwrap/ua-parser-js-0.7.14",
    "whatwg-fetch": "file:node_shrinkwrap/whatwg-fetch-2.0.3"
  }

format 4 is format 3 but with the package.json put back to a more typical format;

  "dependencies": {
    "angular": "1.6.5",
    "asap": "2.0.6",
    "core-js": "1.2.7",
    "create-react-class": "15.6.0",
    "encoding": "0.1.12",
    "fbjs": "0.8.14",
    "iconv-lite": "0.4.18",
    "is-stream": "1.1.0",
    "isomorphic-fetch": "2.2.1",
    "js-tokens": "3.0.2",
    "lodash": "4.17.4",
    "lodash.assign": "4.2.0",
    "loose-envify": "1.3.1",
    "node-fetch": "1.7.2",
    "object-assign": "4.1.1",
    "promise": "7.3.1",
    "prop-types": "15.5.10",
    "react": "15.6.1",
    "setimmediate": "1.0.5",
    "ua-parser-js": "0.7.14",
    "whatwg-fetch": "2.0.3"
  }

Strangely though, this version does not work.

I'll post more as I find it, thanks all.

Answer 23 · 2017-08-18T16:41:40.000Z

Thanks for the continuing investigation and updates!

Answer 24 · 2017-08-20T15:03:03.000Z

@iarna @zkat if you have some time I could use your help if you can please? It's most likely that I have the structure of package-lock.json slightly wrong, but there's also a slim chance this might be an npm bug? I wonder this because format 4 from #83 (comment) is created by installing a local tarball using npm alone, then changing the dependency in package.json back to a semver version number. Again though, this is likely just me misunderstanding something in the new lockfiles.

Thanks as always, grateful for your time and help.

Answer 25 · 2017-08-29T15:13:10.000Z

I've opened npm/npm#18339 over on the npm repo and am available if there's anything at all I can do to help, I know the npm team are crazy busy.

Answer 26 · 2017-08-29T17:54:52.000Z

Yeah, I'll look at this uh, probably next week, but sooner if possible.

Answer 27 · 2017-09-26T11:58:37.000Z

is this something one of us could contribute to do you think @iarna? Or does it look like it could be too involved to be handled outside the core team? if there's any way I can help I'd be happy to. Thanks.

Answer 28 · 2017-10-04T12:48:39.000Z

@iarna Do you happen to have an update? Thanks!

Answer 29 · 2017-10-05T07:29:14.000Z

I wonder if cipm is an alternative route that might work? (https://twitter.com/rebeccaorg/status/902983343376961536). I am away but will try next week if no one beats me to it.

The latest shrinkpack is on the https://github.com/JamieMason/shrinkpack/tree/dev branch and can be run with npm install && npm run build && node dist/bin.js --help

Answer 30 · 2017-10-05T18:35:19.000Z

@timdb Nope! My priorities largely got rejuggled by 2fa, but I'm making sure it's on my list of things.

@JamieMason I don't actually know what the scope of the problem is as yet. If someone wants to take a crack at it, that'd be great though. (And if someone does and has questions about the source, drop by the #npm channel on https://package.community/ and we'd be happy to answer them.)

Answer 31 · 2017-10-19T15:42:24.000Z

Ok, I've gotten some time to look at this, in particular the aggregate-error branch:

There are two problems with the package-lock in that branch.

First, the requires section should still be by version. npm matches them up by comparing values from the requires field with the version field.

Second, the integrity field needs to have hashes from the tar files in your resolved field. Right now it has values from the tgz files that the tar files were derived from.

The actual error itself is coming from extract's subprocesses as they have problems with the shasum failures (as we now extract tarballs in parallel). The aggregate mentioned is the aggregate of the extract workers. It both needs better messaging, well, currently the real errors show up on screen but not in the logs, which is pretty awful.

Answer 32 · 2017-10-19T15:46:00.000Z

Great, thanks @iarna I will take a look at those issues and get them fixed.

Answer 33 · 2017-10-19T15:46:54.000Z

My local testing shows that fixing those won't be quite enough, there's also some underlying npm bug…

Answer 34 · 2017-10-19T16:09:15.000Z

This is my cut-down repro: https://gist.github.com/iarna/a47991307e626ffdef0a3012ae41fae5

Answer 35 · 2017-10-19T16:14:49.000Z

OK thanks for the info, I'll concentrate on correctness of the package lock and head to that channel with any questions, most likely around generating the hashes but will see when I get to it.

Grateful for your help as always.

Answer 36 · 2017-10-19T16:16:27.000Z

ssri is a great way to go generate those hashes =)

Answer 37 · 2017-11-06T15:50:21.000Z

Shrinkpack for npm5 coming soon ? 😃

Answer 38 · 2017-11-10T19:56:24.000Z

Second, the integrity field needs to have hashes from the tar files in your resolved field. Right now it has values from the tgz files that the tar files were derived from.
– @iarna

I just want to check, right now locally I have:

    "add-matchers": {
      "version": "0.5.0",
      "resolved": "file:node_shrinkwrap/add-matchers-0.5.0.tar",
      "integrity": "sha1-UCGQ5HUM1XIWGDkyaLYaFXNm52U= sha512-WqLVrqoSZuHp2fNqq/9XTILkn8VeJCO5QdVvo+qWTAdohrUvtR7sLYA6isVLJsZ2KoU6oRhY7d2yGOdjxrBdpQ=="
    },

This looks like it should be right if I've understood #83 (comment) from @zkat correctly, but is that correct or should it change to contain only the sha for the .tar like so?

    "add-matchers": {
      "version": "0.5.0",
      "resolved": "file:node_shrinkwrap/add-matchers-0.5.0.tar",
      "integrity": "sha512-WqLVrqoSZuHp2fNqq/9XTILkn8VeJCO5QdVvo+qWTAdohrUvtR7sLYA6isVLJsZ2KoU6oRhY7d2yGOdjxrBdpQ=="
    },

Thanks.

Answer 39 · 2017-11-12T19:21:32.000Z

The latest implementation is as follows;

When leaving packages compressed

The integrity properties are left as "<original-tgz-hash>"

When decompressing packages

The integrity properties are set to "<original-tgz-hash> <new-tar-hash>"

Generating hashes

The <new-tar-hash> is generated using the file contents and ssri.stringify

const contents = await fs.readFile(tarPath); 
const integrity = ssri.fromData(contents); 
return ssri.stringify(integrity);

Answer 40 · 2017-11-15T07:54:37.000Z

I just pushed a patch that might help with this -- do you wanna try using npx npmc i? I've published the patch as part of the canary so folks can try it out. It fixed @iarna's repro and a couple of other git/tar/etc-related issues, specially around checksums.

Answer 41 · 2017-11-16T11:16:38.000Z

I've just given this a try @zkat, but as things are at the moment shrinkpack with npm5 does not appear to work.

The gist of the problem is that npmc is still attempting to hit the public registry instead of reading the resolved fields and hitting ./node_shrinkwrap.

The most likely cause is that I don't have the npm-shrinkwrap.json set up exactly as it needs to be.

Here is a script to reproduce:

#!/usr/bin/env bash

rm -rf ~/npm5-and-shrinkpack
mkdir ~/npm5-and-shrinkpack
cd ~/npm5-and-shrinkpack

git clone https://github.com/JamieMason/shrinkpack.git -b dev
cd ~/npm5-and-shrinkpack/shrinkpack
npmc i
npmc run build

cd ~/npm5-and-shrinkpack
npm init -y
npmc install @telerik/eslint-config
npmc install @types/core-js
npmc install @types/jasmine
npmc install ava
npmc install finch

npmc shrinkwrap
node ~/npm5-and-shrinkpack/shrinkpack/dist/bin.js --compress

read -r -p "Disconnect From The Internet Then Hit Enter To Continue" response

npmc cache clear --force
rm -rf node_modules
npmc install

The general format of the npm-shrinkwrap.json produced by shrinkpack is:

{
  "name": "mock-project",
  "version": "1.0.0",
  "lockfileVersion": 1,
  "requires": true,
  "dependencies": {
    // lots of other deps
    "ansi-align": {
      "version": "1.1.0",
      "resolved": "file:node_shrinkwrap/ansi-align-1.1.0.tgz",
      "integrity": "sha1-LwwWWIKXOa3V67FeawxuNCPwFro=",
      "requires": {
        "string-width": "1.0.2"
      }
    }
    // lots of other deps
  }
}

The error log produced by npmc is:

2017-11-16T11_00_30_255Z-debug.log

Notes:

The value in the integrity field (sha1-LwwWWIKXOa3V67FeawxuNCPwFro=) is the original one put there by npmc for ansi-align-1.1.0.tgz, it is the exact same tarball as downloaded from the registry. I this fine or should the sha be regenerated using the tarball on disk?
Is it correct that the requires members have their semver version as the value? or should that be eg. file:node_shrinkwrap/string-width-1.0.2.tgz? I ask because when following the steps at npm/npm#18339 this is what npm itself does to the requires fields but in this discussion here we've talked about it needing to be the semver version.

Thanks all.

Answer 42 · 2017-11-16T11:24:44.000Z

Sorry, I'm also still able to reproduce npm/npm#18339 using npmc.

Answer 43 · 2017-11-17T17:14:13.000Z

I think let's focus on getting @iarna's cut-down repro working in npmc then continue from there.

To save confusion I have left the dev branch at the commit it was at when I wrote #83 (comment) and will do any new work in a new refactor branch for now.

Answer 44 · 2017-11-17T17:43:49.000Z

I've added --loglevel silly to the refactor branch and an example failing test for aggregate error can be seen at https://travis-ci.org/JamieMason/shrinkpack/jobs/303640167.

The dev branch is passing, but I think an important difference between that and the test script I've been using from #83 (comment) is that it does not npm cache clear --force.

I will update the tests to do that as shrinkpack would need to work offline from an empty cache.

Answer 45 · 2017-12-15T14:54:17.000Z

npm 5.6 is out, any news on this?

Answer 46 · 2017-12-15T15:31:12.000Z

This ticket is still blocked by npm/npm#18339.
The npm team are aware of the issue but have a lot on their plate.

For anyone reading this in the future: this is the highest priority issue in shrinkpack and when anything does progress you will hear about it by subscribing. You do not need to poke or nudge this thread to keep it alive (this isn't a dig at you @simllll 😃 👍 )

Answer 47 · 2017-12-15T18:44:23.000Z

It looks like the fix has landed but as far as I can tell it hasn't made it into an npm release yet. Fingers crossed. As always thanks for your continued work on this.

Answer 48 · 2017-12-15T21:12:30.000Z

@rockerest That landed in 5.6.0 which is latest.

Answer 49 · 2017-12-17T13:50:58.000Z

as far as I can tell npm/npm#18339 is still present in 5.6.0, I have left a comment over there. Anyone following this issue might also want to subscribe to that one.

Answer 50 · 2017-12-19T14:54:43.000Z

Spotted this PR over at npm/npm#19423 🎉

Answer 51 · 2018-01-10T12:59:32.000Z

Update On Progress

Applied this patch from npm/npm#19423 to my local installation of npm.
Stopped concatenating hashes in shrinkpack.
Tested it out locally against a project.
It works – npm installs using only npm-shrinkwrap.json and ./node_shrinkwrap and does not use the network.

Outstanding Problem

Caveat: This is pretty complicated and I don't know for certain, but this is how it seems to me – to start off the conversation.

https://github.com/JamieMason/shrinkpack#create-a-project-specific-cache-optional has been a long-standing problem which was possible to work around in older versions of npm, but so far it looks like this workaround no longer works with npm5.

When I'm back on my own machine I will create branches to reproduce from and open an issue over at npm/npm, but for now here is a fairly common use-case which could be affected:

Fork and clone a shrinkpacked repo (let's say that it depends on whatwg-fetch@2.0.3 as an example, just because that has no other dependencies).
npm install it, do some work, submit a PR etc.
When you're done, delete your local clone and fork.
Later, clone some other non-shrinkpacked project that also depends on whatwg-fetch@2.0.3.
Run npm install in this other non-shrinkpacked repo.
npm errors as it can't find /Usr/you/Dev/that-shrinkpacked-repo-from-that-pr-you-did-but-since-deleted/node_shrinkwrap/whatwg-fetch-2.0.3.tar.

The gist is that as npm install has previously been run against a shrinkpacked repo, from then on npm seems to expect that packages previously installed from the file system will still be there.

My instinct is that the local file system shouldn't be treated in the same way as registries are and should be considered unstable and subject to change without notice but @iarna and @zkat know way more than I do about all this, and I'm probably overlooking something.

Secondary problem

Handling git dependencies doesn't quite work yet, this is entirely down to shrinkpack and I'll fix it.

Answer 52 · 2018-01-12T08:51:06.000Z

I don't want to hijack the conversation but how does this currently relate to npm ci (formerly known as cipm)?

Answer 53 · 2018-01-12T09:01:39.000Z

@timdp can you expand on what you mean by how they might relate to each other?

References I know of between shrinkpack and cipm are this comment above when I first heard about cipm (as a note to self mainly) and that zkat mentions shrinkpack in this PR comment in the cipm repo.

Answer 54 · 2018-01-12T13:10:51.000Z

I meant that they're basically trying to achieve the same thing: performing an npm install without network access--or at least that's how I understood it. Hence, I was wondering what the exact case for shrinkpack is versus the one for cipm.

Answer 55 · 2018-01-12T14:33:51.000Z

If npm provide an officially supported form of shrinkpack that would be the ideal, and we'd have no need for shrinkpack.

I asked back in July 2017 whether there was any need for me to add support for npm5 and cipm might be what they had in mind back then but didn't have time yet to get started.

Answer 56 · 2018-01-12T15:22:34.000Z

Okay, but now that cipm is there, is there effectively a difference between cipm with a project-local npm cache (so that you only get the dependencies you need) and shrinkpack?

Answer 57 · 2018-01-15T10:55:45.000Z

I don't know @timdp but from reading https://www.npmjs.com/package/cipm my interpretation is that cipm isn't necessarily for installing offline but is a stricter, more deterministic npm – so would be better-suited to being used with caching proxies or offline mirrors.

If that's correct, then it seems to me that something like shrinkpack could a good fit for preparing an offline mirror which cipm could later install.

Answer 58 · 2018-01-17T11:56:25.000Z

Opened Packages installed from "file:..." become default location for package/version pairs
at npm/npm#19621

Answer 59 · 2018-01-18T03:00:28.000Z

@timdp npm can already do almost fully-offline installs if you have a package-lock.json and a warm cache. This happens -right now-, without any extra flags. The only exception right now is git packages, which need an extra layer of bespoke caching before I can do that with those (though the last real hurdle has been cleared on that).

As far as npm ci goes: @JamieMason's assessment of this is pretty much correct. It works the same as npm i, except it's a bit more strict about a few things and takes advantage of some optimizations specific to CI-like environments. It doesn't change anything major about the installer. You should still be able to use shrinkpack with it.

Thanks all for sticking through this and helping out. I know it's been a while, but I'm looking forward to telling people that they're able to do this, soon! 🎉

Edit: omg this would be a -fantastic- thing to announce in conjunction with the release of npm ci because they're pretty connected! People are gonna be so excited 😻

Answer 60 · 2018-02-18T14:54:55.000Z

There have been some recent developments which can be followed in the #npm channel of package.community at https://discord.gg/RkxC4cA.

The summary is that @zkat has put together patches for shrinkpack, npm, pacote, and cacache to resolve the issues we've been seeing. This means support for shrinkpack is likely to land in one of the next two releases of npm, with shrinkpack 1.x being released around the same time.

Answer 61 · 2018-03-27T09:41:26.000Z

npm v5.8.0 is out - with patches for shrinkpack

Answer 62 · 2018-03-30T09:32:26.000Z

Away at the moment but aiming to work on this Sunday or Monday, thanks all. Almost finally there now I hope.

Answer 63 · 2018-04-02T11:47:35.000Z

.tgz packages seem to work correctly in 5.8.0, they can be installed offline. .tar packages however have some issues and are hitting the network and warning of integrity issues.

The following output is when npm ci is run against the shrinkpack dev branch itself, when it has been bundled using shrinkpack .:

npm ci output

$ npm ci --loglevel http
WARN pacote EINTEGRITY while extracting browserify-zlib@0.1.4 from /Users/foldleft/Dev/shrinkpack/node_shrinkwrap/browserify-zlib-0.1.4.tar.You will have to recreate the file.
WARN pacote EINTEGRITY while extracting @ladjs/time-require@0.1.4 from /Users/foldleft/Dev/shrinkpack/node_shrinkwrap/ladjs-time-require-0.1.4.tar.You will have to recreate the file.
WARN pacote EINTEGRITY while extracting pinkie-promise@1.0.0 from /Users/foldleft/Dev/shrinkpack/node_shrinkwrap/pinkie-promise-1.0.0.tar.You will have to recreate the file.
WARN pacote EINTEGRITY while extracting term-size@1.2.0 from /Users/foldleft/Dev/shrinkpack/node_shrinkwrap/term-size-1.2.0.tar.You will have to recreate the file.
WARN pacote EINTEGRITY while extracting is-promise@2.1.0 from /Users/foldleft/Dev/shrinkpack/node_shrinkwrap/is-promise-2.1.0.tar.You will have to recreate the file.
http fetch GET 304 https://registry.npmjs.org/browserify-zlib 929ms (from cache)0.2 -> /Users/foldleft/Dev/shrinkpack/node_modules/json-parse-better-errors
http fetch GET 200 https://registry.npmjs.org/browserify-zlib/-/browserify-zlib-0.1.4.tgz 57ms (from cache)e_modules/estraverse
WARN pacote EINTEGRITY while extracting symbol-observable@0.2.4 from /Users/foldleft/Dev/shrinkpack/node_shrinkwrap/symbol-observable-0.2.4.tar.You will have to recreate the file.
http fetch GET 304 https://registry.npmjs.org/symbol-observable 65ms (from cache)ldleft/Dev/shrinkpack/node_modules/meow/node_modules/strip-bom
http fetch GET 304 https://registry.npmjs.org/pinkie-promise 547ms (from cache)foldleft/Dev/shrinkpack/node_modules/is-buffer
http fetch GET 200 https://registry.npmjs.org/pinkie-promise/-/pinkie-promise-1.0.0.tgz 43ms (from cache)ode_modules/fill-range
http fetch GET 200 https://registry.npmjs.org/symbol-observable/-/symbol-observable-0.2.4.tgz 152ms (from cache)
http fetch GET 304 https://registry.npmjs.org/is-promise 564ms (from cache) -> /Users/foldleft/Dev/shrinkpack/node_modules/is-equal-shallow
WARN pacote EINTEGRITY while extracting prettier@1.11.1 from /Users/foldleft/Dev/shrinkpack/node_shrinkwrap/prettier-1.11.1.tar.You will have to recreate the file.
WARN pacote EINTEGRITY while extracting pinkie@1.0.0 from /Users/foldleft/Dev/shrinkpack/node_shrinkwrap/pinkie-1.0.0.tar.You will have to recreate the file.
http fetch GET 200 https://registry.npmjs.org/is-promise/-/is-promise-2.1.0.tgz 72ms (from cache)
http fetch GET 304 https://registry.npmjs.org/term-size 539ms (from cache)s/foldleft/Dev/shrinkpack/node_modules/for-in
http fetch GET 200 https://registry.npmjs.org/term-size/-/term-size-1.2.0.tgz 17ms (from cache)
http fetch GET 304 https://registry.npmjs.org/pinkie 69ms (from cache)Users/foldleft/Dev/shrinkpack/node_modules/for-in
http fetch GET 200 https://registry.npmjs.org/pinkie/-/pinkie-1.0.0.tgz 28ms (from cache)
WARN pacote EINTEGRITY while extracting path-parse@1.0.5 from /Users/foldleft/Dev/shrinkpack/node_shrinkwrap/path-parse-1.0.5.tar.You will have to recreate the file.
http fetch GET 304 https://registry.npmjs.org/path-parse 49ms (from cache).11.1 -> /Users/foldleft/Dev/shrinkpack/node_modules/regenerator-runtime
http fetch GET 200 https://registry.npmjs.org/path-parse/-/path-parse-1.0.5.tgz 15ms (from cache)
WARN pacote EINTEGRITY while extracting fsevents@1.1.3 from /Users/foldleft/Dev/shrinkpack/node_shrinkwrap/fsevents-1.1.3.tar.You will have to recreate the file.

> fsevents@1.1.3 install /Users/foldleft/Dev/shrinkpack/node_modules/fsevents
> node install

[fsevents] Success: "/Users/foldleft/Dev/shrinkpack/node_modules/fsevents/lib/binding/Release/node-v57-darwin-x64/fse.node" already installed
Pass --update-binary to reinstall or --build-from-source to recompile
added 445 packages in 5.619s

If I were to have run this while offline, npm would hang during installation.

Answer 64 · 2018-04-02T12:24:58.000Z

Are you sure both of the hashes are listed in integrity? Does it work if you use $ npx my-precious instead of shrinkpack? That might be a good reference for you.

Answer 65 · 2018-04-02T21:41:04.000Z

Out of time for today but I have this branch https://github.com/JamieMason/shrinkpack/tree/npm5 which is a thin wrapper for my-precious. All seems to work as expected but I've yet to get the e2e tests passing. Once I can figure this out I'll have more confidence to release it.

EDIT: Apologies for the email spam subscribers probably got thanks to my repeated pushes to Travis ☝️

Answer 66 · 2018-05-03T14:22:12.000Z

Any progress on this? I don't mind having a look if you're not able to work on it

Answer 67 · 2018-05-03T15:44:34.000Z

Thanks @peteward44 I appreciate the offer.

I have to grudgingly admit that this issue has worn me down and by now I really struggle to find the energy and motivation to get it through. Kat and Rebecca from npm have been amazing and we couldn't ask any more from them, but in two days this issue will be a year old. That's a long slog for one person and over that time each fix has uncovered new issues which took time to resolve before continuing on again.

The under-development version of shrinkpack works, at least on paper, the overall ongoing issue has been around gaining support in npm >=5 for fully offline installations using only a lockfile containing file: paths, as was the case in npm 3 and 4. Again, the npm devs are swamped and it's no simple task.

There is some good news though that npm have announced they will be:

Integrating the functionality of shrinkpack into npm directly. This would allow for entirely offline deploys even without a cache. It will also let you deploy with git dependencies without having git installed on your production machines.
– https://blog.npmjs.org/post/173239798780/beyond-npm6-the-future-of-the-npm-cli

In light of this, I think it would be better to wait until npm archive is released in npm@7. No third party tooling (shrinkpack) will be needed and it's also ridiculously fast compared to shrinkpack.

Sorry everyone for such a long wait, but as a result we will get npm archive so it will be worth it.

Answer 68 · 2018-05-04T07:10:28.000Z

OK fair enough. A shame as shrinkpack is a great tool, but native support for offline installs does make a lot more sense.

Answer 69 · 2018-05-04T07:50:50.000Z

npm taking over is definitely good - giving it a future as first class citizen.

This probably only arrives in half a year or so. In the blog post they state "late summer and fall of 2018" and given they're loaded with work one could guess that the timeline for this feature slips further.

I would welcome a quicker (interim) solution - and it seems we are already very close with shrinkpack working at the core.

For folks who want to help and take over your excellent work to get it finally out of the door - what would be the next steps to be taken?

To me, the next steps look like:

set up & fix tests in the npm5 branch - any new ones?
Report the progress on tests, so that others can chime on problems
Identify potentially further problems in npm
create a pre-release version to gather feedback from users

Answer 70 · 2018-05-04T08:17:10.000Z

@mobidev111 the native npm version is pretty much already implemented and mostly needs attention and polish. You can try it right now with $ npx npmc archive. Regular npm install as of npm@6 should be able to install off that archive. If you want to add/remove packages, though, you'll need to use $ npx npmc add <pkg> and $ npx npmc rm <pkg> so edits to package-lock.json happen correctly.

Answer 71 · 2018-05-04T08:18:03.000Z

The reason it isn't shipped is because I need to take it over the finish line, make sure corner cases are covered, add tests, etc -- but I won't have time allocated to do this for the next couple of months, probably.

Answer 72 · 2018-05-04T08:18:41.000Z

(p.s. $ npx npmc unarchive will return your repo to its original pre-archive format)

Answer 73 · 2018-05-04T10:45:19.000Z

Great news @zkat. I'm not brave enough to use this in my production tool chain just yet, but i look forward to the official release 👍 We've been stuck on npm 4 due to this for a long time now and really can't wait to upgrade

Answer 74 · 2018-05-04T13:26:08.000Z

I have published npm install -g shrinkpack@next which contains the current npm5 branch as 1.0.0-alpha: https://github.com/JamieMason/shrinkpack/blob/npm5/CHANGELOG.md.

It is a thin wrapper for my-precious rather than the ongoing work I did in the dev branch, I expect there may be issues but please let us know how it goes for you 👍

Thanks all for your comments, @mobidev111 thanks a lot for that useful comment and organisation.

Answer 75 · 2018-05-16T13:17:28.000Z

Hi @JamieMason,

Thanks for all your work, I know firsthand how building an open source project can be disorienting and tiring, and I just wanted to say that your efforts are very much appreciated and needed!

Congrats on getting your code into npm archive!

Answer 76 · 2018-05-22T14:57:13.000Z

Hi. @JamieMason
I'm getting the infamous

[ERROR] npm WARN optional SKIPPING OPTIONAL DEPENDENCY: aggregate error
[ERROR]
[ERROR] npm ERR! aggregate error

when running npm install

I'm using node v9.11.1 / npm 5.6.0 and shrinkpack 1 - alpha.
Funky enough, it only happens on the Jenkins Linux machine.
I've tried running npm i against the archived-packages locally, with the internet connection turned off, and no problem whatsoever occured.

Answer 77 · 2018-08-17T15:23:06.000Z

npm/cli#1 got closed down :(
Looks like yarn is the only solution, other than staying with vastly slower npm v4

Answer 78 · 2018-08-17T18:01:08.000Z

@themoonrat we're looking into ways to reintegrate this and see where it fits into our product plans for the CLI. In the meantime, the core support for this kind of thing did land in npm. Any tool that does the same kind of rewriting should provide similar support. This means you can use $ npx my-precious or shrinkpack can be updated to do things with libprecious itself, and it should be way faster, and npm will do installations off the local tarballs very very quickly. :) I'm also still around to help integrate things if shrinkpack is interested in moving forward with this while we figure out how and whether to land this in npm proper.

Answer 79 · 2020-08-04T10:00:51.000Z

So it's been while.
I'm still using shrinkpack and npm@4 since it still seems to be the only way to do fully offline installs without zipping up the node_modules folder but it is steadily becoming problematic to be on such an old version.
Has there been any movement on getting npm able to install on an offline air-gapped machine or with shrinkpack using newer versions of npm?

Answer 80 · 2020-08-06T11:28:00.000Z

Hey @thomashardwick, check out #83 (comment) as I think it's still about right as a summary of where shrinkpack is at. Quite some time has passed though and I've lost touch with what npm provides out of the box in this area, there was talk of an npm archive command but again I don't know the latest.

Answer 81 · 2020-08-07T07:23:50.000Z

@thomashardwick I know this is a very different solution, but have you tried something like https://www.snowpack.dev/ or my own much simpler (better, in my opinion) https://www.npmjs.com/package/@thomasrandolph/icepack?

In both cases, instead of saving a local copy of the full dependency tree, those tools build a bundled copy of the appropriate dependencies. You would do that once, while you have an internet connection. Then, all future references to the dependency just reference the bundled file. One key part of both of these tools is the bundled version is always browser-safe, too, so you can run either tool ahead of time and then you have a set of dependencies that you never have to touch again, including compiling (if you ship ES6 modules to the browser).

Again, I know this is a really big deviation from shrinkpack, but I've found it's been extremely helpful to distance myself as far as possible from the npm ecosystem.

Answer 82 · 2020-08-07T10:37:10.000Z

I've since changed to use Yarn and it's offline mirror, which works pretty much like shrinkpack does.

https://classic.yarnpkg.com/blog/2016/11/24/offline-mirror/

It would seem NPM has lost interest in allowing users to offline dependencies as that goes against it's business model. A shame

Answer 83 · 2020-08-07T10:44:51.000Z

Keep an eye on yarnpkg/yarn#541 @peteward44 as Yarn's offline mirror can gradually harm the performance of your Git repo over time.

Answer 84 · 2020-08-08T04:34:59.000Z

I've got no idea what npm's plans are these days, but they do have an active rfc process going, and it may be useful to bring this up there again.

Answer 85 · 2021-12-29T22:10:50.000Z

shrinkpack@0.19.0 has been released which adds support for npm 7 and up. Thanks again to Kat and Rebecca for all of your help during your time at npm ❤️ .

Answer 86 · 2021-12-30T21:32:33.000Z

This is amazing