Missing dependencies after running `npm install` a second time

Question

Missing dependencies after running `npm install` a second time

schmod opened this issue 7 years ago · 64 comments

Answer 1 · 2017-05-31T19:42:48.000Z

Some quick discussion:

The broken transitive dependency (kue) referenced in the reproduction steps above was resolved through a GitHub dependency (foundry-kue-scheduler depends on a GitHub-hosted fork of kue-scheduler). I haven't been able to identify if that's the cause of this regression, but it seems like a potential culprit.

The fact that the "bare" npm install command could modify an existing package-lock.json was surprising to me.

I would not be surprised if npm prune, npm update, or npm install <package> modify the lockfile, but definitely was not expecting npm install to make changes on its own.

This seems like a bug, but there doesn't seem to be much documentation about the contracts that NPM makes with regards to lockfiles, or how they're meant to be used in a typical workflow. For all I know, this could be an expected behavior. More documentation would be helpful.

My $0.02 is that the "bare" npm install command should strive to be idempotent, and should not make any changes (outside of node_modules or caches) when a lockfile is present.

Answer 2 · 2017-05-31T19:49:14.000Z

Dropping a link to #16837 here, because it seems similar.

Answer 3 · 2017-05-31T21:30:40.000Z

Seeing these same results. We committed our package-lock.json after a brand new npm i and then on Jenkins we (running same node version, same npm version) see missing module errors. I also see the same results where the package-lock.json gets weird and smaller after every npm i run.

By weird, I'm seeing an issue where the repository URL ends up in the "version" field of package-lock.json.

Answer 4 · 2017-05-31T21:34:17.000Z

And to confirm I also see this only when locally removing the node_modules folder (which is also the environment in CI)

Answer 5 · 2017-05-31T23:59:35.000Z

I am seeing something very similar when updating a single package. For example, once everything is working fine, I can do this:

➜  models (master) ✔ npm i
up to date in 1.561s
➜  models (master) ✔ rm -rf node_modules/plump 
➜  models (master) ✔ npm i
added 1 package and removed 4 packages in 1.792s
➜  models (master) ✗

note that I just deleted one package that I directly depend on, and then running npm i a second time correctly installs the deleted package, but seems to delete the packages that were depended on by that package I manually deleted. So now (in this case), plump is properly installed, but rxjs and merge-options are both missing (both are in the dependencies entry for plump). I'm not entirely sure where the 4 removed packages total comes from - I assume they are symbol-observable and is-plain-obj, which are the sole declared dependencies of rxjs and merge-options respectively.

What is weirder is that if I do npm i --no-save rxjs, I get back "added 2 packages", but they don't actually add (npm ls rxjs shows it's not there, and just doing an ls -laR node_modules | grep rxjs confirms this).

If I do something silly like go into a working directory that's all properly installed and do npm i --no-save rxjs (or some other package that is a declared dependency of the thing I'm working on but was already installed), it goes nuts - in this case it removed 123 packages and updated 211 packages, when in fact it should have done nothing (rxjs was already installed and up to date).

In all cases, this is only repaired by deleting both package-lock.json and node_modules and doing a fresh npm i.

Answer 6 · 2017-06-02T01:11:29.000Z

I was going to file my own bug, but I think this one covers my experience. I've managed to create a minimal experiment and demo repo that reproduces this behavior; the README has all the details:

https://github.com/mbland/npm-v5-package-lock-bug-demo

Hopefully it's helpful here. The upshot: the problem appears related to devDependency packages whose own dependency specs are set tolatest (at least, that's how it appeared in my project with a devDependency on live-server). It also only appears to manifest when the original node_modules directory is removed and recreated.

Answer 7 · 2017-06-02T02:13:30.000Z

@mbland awesome work! Thanks for this 🎉

Answer 8 · 2017-06-04T12:02:10.000Z

FYI, just re-ran all the scenarios with npm v5.0.2, and got essentially the same results. The only difference is that buffer-shims is no longer installed, because readable-stream v2.2.10 no longer depends on it.

Answer 9 · 2017-06-05T21:33:53.000Z

I think I was able to work around it by manually installing the missing dependencies until there were none left and letting npm update the package-lock.json as I go along.

For example:

❯ npm test

module.js:487
    throw err;
    ^

Error: Cannot find module 'pretty-error'
    at Function.Module._resolveFilename (module.js:485:15)

❯ npm install pretty-error
npm WARN solution-manager@1.0.0 No repository field.

+ pretty-error@2.1.0
added 21 packages, removed 1 package and updated 14 packages in 3.072s

❯ npm run test

module.js:487
    throw err;
    ^

Error: Cannot find module 'wrap-ansi'
    at Function.Module._resolveFilename (module.js:485:15)

❯ npm install wrap-ansi
npm WARN solution-manager@1.0.0 No repository field.

+ wrap-ansi@2.1.0
added 1 package, removed 1 package and updated 14 packages in 2.519s

❯ npm test

IT WORKS!

And npm updated both package.json and package-lock.json.

Now to confirm it actually helped anything:

❯ rm node_modules

❯ npm test

module.js:487
    throw err;
    ^

Error: Cannot find module 'yargs'
    at Function.Module._resolveFilename (module.js:485:15)

❯ npm install

added 380 packages in 55.731s

❯ npm test

IT WORKS!

Looking good!

I'm going to do some more testing, however I wanted to report my strange findings here in case it helps anyone. Hope we can get this resolved soon!

Answer 10 · 2017-06-05T21:38:24.000Z

I can confirm the comment by @Glavin001. We handled it by having our developers that removed node_modules remove package-lock.json before running npm install and as long as npm produced package-lock.json rather than pulling from a git repo subsequent npm install commands did not modify the lock file.

Answer 11 · 2017-06-08T08:39:33.000Z

I upgraded to npm5 while using Expo (for react native, it installs about 700 packages within node_modules without adding them to package.json)

I did npm install for the first time to add one package and it pruned the entire node_modules directory, and I had to downgrade to npm4.x , create a new Expo project, and copy paste my src folder and configs.

Is it a feature that npm5 prunes by default on each install or was it an attempt to make it "better than yarn"? Because that didn't work at all, I went back to yarn after all.

Answer 12 · 2017-06-10T15:03:18.000Z

The behavior of the mbland/npm-v5-package-lock-bug-demo scenarios still holds under 5.0.3. (Note: I haven't been pushing package-lock*.json updates, as they only reflect updated package versions, not fundamental changes in npm behavior.)

Though I've a lot of detail in the README of that repo already, I've a few more observations and thoughts I'll share here.

Problems arise when only one artifact is present

It seems npm v5 gets into trouble when one of package-lock.json or node_modules is present, but the other is missing. Upgrading from a previous npm version to npm v5 and running npm install within an existing working directory would appear to fit this pattern (i.e. node_modules is present, package-lock.json is missing). As one small data point, I can confirm that adding and removing http-server multiple times as a devDependencies package in my demo repo after starting from a clean state (where both package-lock.json and node_modules are generated for the first time) behaves as expected.

First problem: missing `package-lock.json` results in `version` values becoming URLs

As I noted in my repo's README, in the case where node_modules is present and package-lock.json is regenerated, running npm install results in some version strings becoming URLs instead of semver specs (from the fsevents entry from package-lock.json):

--- package-lock-00.json	2017-06-10 14:59:58.000000000 +0200
+++ package-lock-01.json	2017-06-10 15:04:20.000000000 +0200
@@ -263,688 +263,688 @@
       "optional": true,
       "dependencies": {
         "abbrev": {
-          "version": "1.1.0",
-          "bundled": true,
+          "version": "https://registry.npmjs.org/abbrev/-/abbrev-1.1.0.tgz",
+          "integrity": "sha1-0FVMIlZjbi9W58LlrRg/hZQo2B8=",
           "dev": true,
           "optional": true
         },
         "ansi-regex": {
-          "version": "2.1.1",
-          "bundled": true,
+          "version": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-2.1.1.tgz",
+          "integrity": "sha1-w7M6te42DYbg5ijwRorn7yfWVN8=",
           "dev": true
         },
         "ansi-styles": {
-          "version": "2.2.1",
-          "bundled": true,
+          "version": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-2.2.1.tgz",
+          "integrity": "sha1-tDLdM1i2NM914eRmQ2gkBTPB3b4=",
           "dev": true,
           "optional": true
         },
[ ...snipped... ]

This screws with some packages such as istanbul-lib-instrument, which performs computations based on the version property.

Would it be worthwhile to have npm install remove the old node_modules and regenerate both in this case? It's slightly annoying, but not as annoying as having some packages fail with cryptic errors.

Major problem: missing `node_modules` and dependencies of devDependencies with a version spec of `latest`

Also, as I mentioned before in my repo's README, the major problem of packages being removed seems to affect packages with a version spec of latest that appear as dependencies of packages from the project's top-level devDependencies list. (And, of course, dependencies of those packages, and so on.) At least, that's what I've observed so far; perhaps others can confirm whether this constraint holds in their projects, or if it happens to other packages that don't fit this pattern.

Here's a demonstration of what I observe after removing node_modules and running npm install twice:

$ cd node_modules
$ jq '.dependencies' live-server/package.json 
{
  "chokidar": "^1.6.0",
  "colors": "latest",
  "connect": "3.5.x",
  "cors": "latest",
  "event-stream": "latest",
  "faye-websocket": "0.11.x",
  "http-auth": "3.1.x",
  "morgan": "^1.6.1",
  "object-assign": "latest",
  "opn": "latest",
  "proxy-middleware": "latest",
  "send": "latest",
  "serve-index": "^1.7.2"
}

$ jq -r '.dependencies | keys | @sh' live-server/package.json | xargs ls -d                 
ls: colors: No such file or directory   
ls: cors: No such file or directory     
ls: event-stream: No such file or directory                                     
ls: object-assign: No such file or directory                                    
ls: opn: No such file or directory      
ls: proxy-middleware: No such file or directory                                 
ls: send: No such file or directory     
chokidar       connect        faye-websocket http-auth      morgan         serve-index

`_requiredBy` becomes empty

Below is a diff for node_modules/colors/package.json, a live-server dependency with a version spec of latest, generated within mbland/npm-v5-package-lock-bug-demo.

The first version (-) is from a clean install where npm install generates both package-lock.json and node_modules.
The second version (+) is generated from the first run of npm install when package-lock.json is present, but node_modules is not; when npm install is run once more, the package disappears.

What appears most significant to me about the diff is that:

_requiredBy becomes empty in the second case (+), even though it's listed in the dependencies section of node_modules/live-server/package.json across all npm install runs
_where is shortened from ../npm-v5.0.1-package-lock-bug-demo/node_modules/live-server to ../npm-v5.0.1-package-lock-bug-demo
other live-server dependencies that have regular semver specs, though their package.json files demonstrate similar changes, are not removed from node_modules in subsequent runs.

diff -uNr node_modules-00/colors/package.json node_modules/colors/package.json
--- node_modules-00/colors/package.json 2017-06-10 15:25:40.000000000 +0200
+++ node_modules/colors/package.json  2017-06-10 15:26:12.000000000 +0200
@@ -1,35 +1,36 @@
 {
-  "_from": "colors@latest",
+  "_args": [
+    [
+      "colors@1.1.2",
+      "/Users/msb/src/mbland/npm-v5.0.1-package-lock-bug-demo"
+    ]
+  ],
+  "_from": "colors@1.1.2",
   "_id": "colors@1.1.2",
   "_inBundle": false,
   "_integrity": "sha1-FopHAXVran9RoSzgyXv6KMCE7WM=",
   "_location": "/colors",
   "_phantomChildren": {},
   "_requested": {
-    "type": "tag",
+    "type": "version",
     "registry": true,
-    "raw": "colors@latest",
+    "raw": "colors@1.1.2",
     "name": "colors",
     "escapedName": "colors",
-    "rawSpec": "latest",
+    "rawSpec": "1.1.2",
     "saveSpec": null,
-    "fetchSpec": "latest"
+    "fetchSpec": "1.1.2"
   },
-  "_requiredBy": [
-    "/live-server"
-  ],
+  "_requiredBy": [],
   "_resolved": "https://registry.npmjs.org/colors/-/colors-1.1.2.tgz",
-  "_shasum": "168a4701756b6a7f51a12ce0c97bfa28c084ed63",
-  "_spec": "colors@latest",
-  "_where": "/Users/msb/src/mbland/npm-v5.0.1-package-lock-bug-demo/node_modules/live-server",
+  "_spec": "1.1.2",
+  "_where": "/Users/msb/src/mbland/npm-v5.0.1-package-lock-bug-demo",
   "author": {
     "name": "Marak Squires"
   },
   "bugs": {
     "url": "https://github.com/Marak/colors.js/issues"
   },
-  "bundleDependencies": false,
-  "deprecated": false,
   "description": "get colors in your node.js console",
   "engines": {
     "node": ">=0.1.90

Conclusion

Right now the easiest workaround appears to be removing both artifacts and letting npm generate both—but checking package-lock.json into the repo and generating node_modules from it is the whole point. Currently, npm install does seem to install all the right packages and versions from package-lock.json the first time you run npm install (so Travis is usually happy), but it behaves strangely every time after that until you remove both artifacts and start from scratch.

Hopefully some of these clues help point to the source of the problem.

Answer 13 · 2017-06-12T20:05:45.000Z

In all seriousness, I would advise anybody pursuing the workarounds proposed in this thread to downgrade to npm v4 for the time being.

Answer 14 · 2017-06-12T20:57:16.000Z

Same here, downgrading to v4 made my life seem manageable again.

Answer 15 · 2017-06-13T13:13:24.000Z

Or, use yarn... 😬

Answer 16 · 2017-06-13T22:18:34.000Z

Also experiencing this issue on node 8.1.0 / npm 5.0.3 - and I've made a new, minimal test reproduction case:
https://github.com/js-n/app-a is an (empty) app which has a git dependency on https://github.com/js-n/dep-a which has a git dependency on https://github.com/js-n/dep-b.

By looking at the commit history on https://github.com/js-n/app-a you can see the changes that occur. When there is no existing package-lock.json, all of the transitive dependencies are properly installed and present in the package-lock.json.

Then, after removing node_modules and re-running npm i, the transitive dependencies are not installed in node_modules and are no longer present in package-lock.json.

Answer 17 · 2017-06-14T16:20:55.000Z

here is another minimal way to reproduce the bug:

npx npm@5.0.3 init -y
npx npm@5.0.3 install
# manually add this dependency (not using `npm i`):
# "dependencies" : { "options": "latest" }
npx npm@5.0.3 install

At the end of the second install, the options dependency will still be missing.

Answer 18 · 2017-06-15T15:32:14.000Z

I stumbled upon this bug today, using react-native-cli I did :

node version 8.1.2, npm 5.0.3

react-native init PROJECT_NAME
cd into project and installing a new package using npm install --save

The out was :

+ react-navigation@1.0.0-beta.11
added 8 packages, removed 664 packages and updated 17 packages in 9.567s

It removed 664 (!!!) packages ... To fix it I simply re-run npm install after deleting the node_modules/ directory

Answer 19 · 2017-06-15T17:14:49.000Z

Much like @MehdiAlouafi I have a bunch of local packages, all marked private: true, not from the npm registry, which are being removed. Luckily they're in source control so I can reinstall them easily. But if I made a local change, didn't check the changes in, then installed another packages, npm install would cause data loss. Which is a bit worrying.

Answer 20 · 2017-06-16T06:05:32.000Z

This probably won't work for all contexts, but I was able to work around this issue by adding package-lock=false to ~/.npmrc in my docker image prior to running (yo|init|install) the first time.

Answer 21 · 2017-06-16T07:05:00.000Z

As workaround, you could run npm shrinkwrap after fresh install. Npm doesn't create lock file if shrinkwrap already exists.

Answer 22 · 2017-06-28T07:07:30.000Z

I have same problem.

node -v
v6.11.0
npm -v
5.0.4

Answer 23 · 2017-06-28T16:06:44.000Z

Dropping by for a quick update: I got from vacation just yesterday, and @iarna was sick for a while. This bug is top of the list to fix and we're currently working on it. <3

Answer 24 · 2017-06-30T21:57:20.000Z

Another update: this will be fixed by #16839 and is already available in the canary, npmc@5.0.4-canary.11.

The patch will go into the next release. Thanks, y'all!

Answer 25 · 2017-07-01T04:22:29.000Z

Gonna close this, since it landed in release-next.

It should be available next Wednesday in mainline. You can use the canary until then!

Answer 26 · 2017-07-05T09:37:16.000Z

Thumbs up!!! Great job!

Answer 27 · 2017-07-06T14:55:32.000Z

Huge thanks to @zkat and @iarna!!

Answer 28 · 2017-07-06T18:58:00.000Z

Yes... Thanks for the quick turn around @zkat and @iarna .

I'm still noticing some weirdness though. :(
I'm using version 5.1.0 and when I install a fresh module...

npm i -D gulp-uglify

The appropriate line appears in the package.json file. Yet there is no actual package in the node_modules directory.

I tried removing the lock file and it still did the same thing.

Answer 29 · 2017-07-06T20:30:21.000Z

Hi @popkinj - since this issue has been closed and is already pretty long, it can be difficult to follow with new information. You might have better luck opening a new issue with detailed reproduction steps.

Answer 30 · 2017-07-06T21:13:26.000Z

Will do... Thanks @js-n .

Answer 31 · 2017-07-19T14:05:09.000Z

I urge the team to update Node.js to v8.1.5 that'd ship with NPM v5.3.0 as currently Node.js v8.1.4 is shipping with a "broken" NPM (v5.0.3)

This bug is especially notorious because due to the fact it's hard to Google (No proper results for "Cannot find module '' ") and this issue is closed and hard to find under a myriad of other issues makes for an especially bad experience to anyone using NPM frequently updating to Node v8

Thank you!

Answer 32 · 2017-07-25T09:43:15.000Z

Still facing this issue with npm@5.3.0 and happy to found this issue listet here since i already was running mad....

Which version should contain the fix? 5.3.0 has not apparently

Answer 33 · 2017-07-25T09:54:50.000Z

Yep, I am having the same issue too, even on npm@5.3.0 :(

Answer 34 · 2017-07-25T16:51:52.000Z

@itinance @binarykitchen it would be most helpful if you could include specific reproduction steps for what you're seeing in a new issue. Since this issue was closed, many people have reported that their issues have been resolved and npm@5.3.0 is working correctly. This means you might be experiencing new issues, or under slightly different circumstances. This warrants opening a new issue for proper investigation :)

Answer 35 · 2017-07-25T19:40:39.000Z

@js-n I know this isn't incredibly detailed, but I just had this happen with 5.3.0:

Working dependencies
Went to update a dependency using npm install react-virtualized@latest --save

+ react-virtualized@9.9.0
added 6 packages, removed 154 packages and updated 2 packages in 19.285s

It removed react-transition-group which is in my package.json (among others, it seems)

Answer 36 · 2017-07-25T22:20:43.000Z

@js-n You're right. Here an example. I go into a running app with multiple dependencies (react-native). Now i install a new library:

1. Before "npm install":

$ du -sh node_modules/
128M	node_modules/

$ ls -l node_modules/react-native
total 88
-rw-r--r--   1 hagen  wheel  1533 Jul 17 23:00 LICENSE
drwxr-xr-x  45 hagen  wheel  1598 Jul 26 00:22 Libraries
-rw-r--r--   1 hagen  wheel  1989 Jul 17 23:00 PATENTS
drwxr-xr-x  13 hagen  wheel   476 Jul 26 00:22 React
-rw-r--r--   1 hagen  wheel  7719 Jul 17 23:00 React.podspec
drwxr-xr-x   4 hagen  wheel   374 Jul 26 00:22 ReactAndroid
drwxr-xr-x   6 hagen  wheel   238 Jul 26 00:22 ReactCommon
drwxr-xr-x   3 hagen  wheel   102 Jul 26 00:22 android
-rw-r--r--   1 hagen  wheel   372 Jul 17 23:00 cli.js
drwxr-xr-x   2 hagen  wheel   442 Jul 26 00:22 flow
drwxr-xr-x   2 hagen  wheel   170 Jul 26 00:22 jest
-rw-r--r--   1 hagen  wheel   696 Jul 17 23:00 jest-preset.json
drwxr-xr-x   2 hagen  wheel   408 Jul 26 00:22 lib
drwxr-xr-x  20 hagen  wheel   918 Jul 26 00:22 local-cli
drwxr-xr-x  12 hagen  wheel   408 Jul 26 00:22 node_modules
-rw-r--r--   1 hagen  wheel  7738 Jul 17 23:00 package.json
-rw-r--r--   1 hagen  wheel  5048 Jul 17 23:00 react.gradle
drwxr-xr-x   2 hagen  wheel   272 Jul 26 00:22 scripts
-rw-r--r--   1 hagen  wheel  1508 Jul 17 23:00 setupBabel.js
drwxr-xr-x   2 hagen  wheel   170 Jul 26 00:22 third-party-podspecs

2. "npm install":

$ npm install react-native-video-processing --save
npm WARN gentlyRm not removing /data/workspace/cbl/video_local/node_modules/.bin/jest as it wasn't installed by /data/workspace/cbl/video_local/node_modules/jest
npm WARN gentlyRm not removing /data/workspace/cbl/video_local/node_modules/.bin/uglifyjs as it wasn't installed by /data/workspace/cbl/video_local/node_modules/uglify-js
npm WARN gentlyRm not removing /data/workspace/cbl/video_local/node_modules/.bin/sane as it wasn't installed by /data/workspace/clb/video_local/node_modules/sane
npm WARN gentlyRm not removing /data/workspace/cbl/video_local/node_modules/.bin/uuid as it wasn't installed by /data/workspace/cbl/video_local/node_modules/uuid
npm WARN gentlyRm not removing /data/workspace/cbl/video_local/node_modules/.bin/json5 as it wasn't installed by /data/workspace/cbl/video_local/node_modules/json5
npm WARN gentlyRm not removing /data/workspace/cbl/video_local/node_modules/.bin/esparse as it wasn't installed by /data/workspace/cbl/video_local/node_modules/esprima
npm WARN gentlyRm not removing /data/workspace/cbl/video_local/node_modules/.bin/esvalidate as it wasn't installed by /data/workspace/Cooblr/video_local/node_modules/esprima
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN react-native-video-processing@1.12.5 requires a peer of react-native@>=0.35 but none was installed.

+ react-native-video-processing@1.12.5
added 26 packages, removed 585 packages and updated 82 packages in 14.333s

The disk usage of "node_modules" is after this command only 33 MB. Formerly it was up to 120 MB.

$ du -sh node_modules/
 33M	node_modules/

"npm install" removed nearly all dependencies.

The removed dependencies are looking like this now:

$ ls -lr node_modules/react-native
total 0
drwxr-xr-x  3 hagen  wheel  102 Jul 26 00:16 node_modules

Versions:
$ npm -v && node -v
5.3.0
v6.2.2

I can replay this all the time.

Answer 37 · 2017-07-25T22:46:49.000Z

@itinance this is a useful reproduction case. I don't work at npm, but you might have more luck getting this in front of an npm developer if you open a new issue with this information, since this issue has been closed for a while now

Answer 38 · 2017-07-25T23:12:59.000Z

@js-n thx for your advice. did it here #17929

Answer 39 · 2017-07-31T11:19:15.000Z

Have the same problem with npm 5.3.0 upgraded from project that used previous version of npm.

Answer 40 · 2017-08-10T17:22:13.000Z

Can we reopen this issue or open a new ticket? It's clearly still a problem. Came here to try to understand why NPM is straight up not installing certain dependancies the first time (and every other time) we setup a repo. It's messing with our deployment process. With the following command, I am able to get it failing in every other build:

rm -fr node_modules/; npm install; npm run watch

@zkat

Answer 41 · 2017-08-14T06:47:34.000Z

I'm still hitting this also 😭 I keep having to nvm use v7.10 and npm install to work around it. Anything on the 5.0 branch is still causing packages to be missing from "node_modules".

Affected project, if you want to try it out: http://github.com/w3c/respec/

Answer 42 · 2017-08-15T02:15:29.000Z

A new bug was opened at #17929. Unfortunately, it doesn't appear to have been triaged.

Answer 43 · 2017-08-16T18:40:01.000Z

Do you have a package-lock.json file left over from v5.0.0?

IIRC, this bug was also causing packages to be removed from package-lock.json in addition to being physically removed from node_modules.

Answer 44 · 2017-08-16T20:53:29.000Z

@schmod That's interesting. Disclaimer: I do, yes, understand how paths work and that different packages are exactly that, different. That said... would nvm or any other similar that has package-lock.json hanging around potentially interfere in this process in any way, too?

I'm asking because your comment inspired me to updatedb and then search my current files for anything named package-lock.json. nvm popped up because that was the first (highly uninformed) method I unsuccessfully tried a few days ago before eventually installing nodejs.

Mine appears to "just" be sitting as /src, but maybe something I did as an inexperienced newbie crosswired things a couple days ago.... or not. 😄

Answer 45 · 2017-08-18T23:03:52.000Z

I wasted so many hours today chasing down issues that ended up leading me to this thread pointing to issues with npm 5.3. I was already targeting 5.3 b/c of issues with 4.X!
As soon as I replaced all my calls to npm install with yarn equivalents my build work great! Feels much like the first time yarn came out and was a dramatic improvement.

Answer 46 · 2017-08-19T03:40:43.000Z

I know this bug can be super frustrating, but please kindly let's not turn this into a yarn vs npm flame war. Doing so risks getting this critical issue getting closed and ignored.

Answer 47 · 2017-08-20T12:06:16.000Z

Competition can promote progress, and under the influence of each other I expect them get better and better. Create more useful features. When npm or yarn take some bug like this, you don't need angry. use the other one to make your job work. Then~~~~

Don't worry be happy. enjoy your life. I'm sure the future version of them will be better

Answer 48 · 2017-09-19T17:59:05.000Z

Any news on this? npm install's consistently remove required modules and forces me to nuke package-lock and node_modules and install anew, around 80-90% of the time.

Answer 49 · 2017-10-11T21:26:42.000Z

I'm so tired about this have the same issue make npm install and for some reason it remove 1425 packages and add 2916 instead of install one with dependencies. Maybe npm i have some key like --do-not-remove-packages or --do-not-make-me-cry

Answer 50 · 2017-10-15T01:36:52.000Z

If anyone comes across this issue I just did brew install node and brew install npm, linked and overwrote, and was able to continue after that. I'm on a machine I haven't worked on in over a year though so YMMV

Answer 51 · 2017-10-26T22:58:57.000Z

I was having what seemed like a related issue to this (but without a lock file) and worked around it by running 'npm install', then 'npm prune', and then 'npm install' again.

Answer 52 · 2018-01-10T11:19:38.000Z

Hi there,
with npm@5.5.1 I'm still having this issue with my git+ssh module. To reproduce:
package.json (where the version for moduleC is the same for both dependencies)

moduleA
    - git+ssh://git@bitbucket.organization/moduleC.git
moduleB
    - git+ssh://git@bitbucket.organization/moduleC.git

npm install (here the package-lock.json does not exist)

node_modules/
    - moduleA
    - moduleB
    - moduleC

rm -rf node_modules
npm install (here the package-lock.json exists)

node_modules/
    - moduleA
         - moduleC
    - moduleB
         - moduleC

Is it a known bug?

Mirko

Answer 53 · 2018-01-11T02:34:29.000Z

@mirkods note that 5.6.0 is now the current stable, so it might be better if you re-test with that

Answer 54 · 2018-01-11T07:27:30.000Z

Hi @jokeyrhyme I just tested with npm@5.6.0 and I can confirm that the issue still exists in the latest version

Answer 55 · 2018-01-11T10:09:57.000Z

Yes, this problem still exists in 5.6.0 when:

"dependencies": {
    "aRepo": "git+ssh://url/aRepo.git"
}

then aRepo and the dependencies of aRepo are removed after an npm install.

Answer 56 · 2018-01-17T01:43:15.000Z

I can also confirm this - using private repo aka git+ssh act very strange.
After runing npm update, sometimes it disappears for some reason although It appeared for a moment during instalation. I am not able to say what cause that but it act totaly random.

Is it possible that during typing keyphrase for ssh, the time used to do it can change anything?

Answer 57 · 2018-01-23T13:48:09.000Z

Running into a similar issue with npm prune --production.

A git+ssh dependency that is defined in dependencies disappears after npm prune --production.

Answer 58 · 2018-03-05T03:11:24.000Z

Same problem.

After installing dependency, like "npm install -D $depname", gulp appears missing. Gulp is defined if package.json as "github:gulpjs/gulp#4.0". "npm install" then fixes the problem.

Answer 59 · 2018-03-10T22:36:26.000Z

Same here on npm v5.7.1 with cyrillic-to-translit-js being defined in package.json as

    "cyrillic-to-translit-js": "github:greybax/cyrillic-to-translit-js#master"

Seems the issue is still relevant. @zkat mind reopening?

Answer 60 · 2018-03-21T20:36:47.000Z

This issue happened to me when I requested from the Development server a bundle without the platform and dev flags.

I was requesting the bundle like http://192.168.192.30:8081/index.bundle

Once I changed it to http://192.168.192.30:8081/index.bundle?platform=ios&dev=true it worked well again.

Answer 61 · 2018-03-29T17:13:49.000Z

I've just seen this on a Dependabot pull request and dug into it. I think I have a fix, but my knowledge of the npm codebase / JS isn't good enough to implement it and (particularly) write tests for it.

First up, here are some reproduction steps:

# Get set up with a problematic update
git clone https://github.com/GilbertGobbels/GAwesomeBot.git
cd GAwesomeBot
git checkout indev-4.0.2

# Generate a bad package-lock.json
npm install null@0 --package-lock-only # null@0 could be anything here

git diff
# Gaze in horror at all the git-source dependencies having been removed (example: canvas-prebuilt)

The --package-lock-only modifier isn't necessary in the above (it just speeds things up), and the argument npm@5.8.0 could be anything - the problem is caused by passing an argument

Next up, I think I've isolated the problem in npm:
The cause of the above is here. The top part of the conditional is only run if an argument is passed, but the problem is that these lines are only run if an argument isn't passed. For whatever reason, git dependencies don't make it into the tree unless they're loaded.

Finally here is a PR that I'm pretty sure fixes this
I've put the above together into a PR at #20198, but JavaScript isn't my strongest language and it could really do with a test. There might also be performance optimisations available that I'm not thinking about (and were the reason for the conditional to start with, although this seems unlikely given the codepath).

I'd love help from a maintainer to get it tested and over the line.

Answer 62 · 2018-04-23T03:13:33.000Z

The same problem!

What is the need for npm install something being so destructive ?

Answer 63 · 2018-04-25T14:19:43.000Z

Well, I've found a reason of this "remove all of them" behavior of npm.

It's all about package.json file and dependencies / devDependencies fields.

In the process of installation of ANY package locally, npm will auto-scan versions of every package recorded in the package.json file, and will compare all of them with versions of packages that you have in node_modules folder. And, in case when you take someone's app (with package.json file, where recorded versions ^4.x.x) and leave your node_modules folder with newer (versions 5.x.x) packages - npm will delete all the packages, that didn't match one with other.

I will just remind, that version ^4.x.x does not match with 5.x.x, because this triangle caret ^ takes "upper effect" only on minor and patch numbers of version. (https://bytearcher.com/articles/semver-explained-why-theres-a-caret-in-my-package-json/)

So, I think it can be confusing for a lot of people. When, for example they know, that Angular 5 doesn't has "cataclysmic" changes in compare to Angular 4.
But "wise" npm thinks otherwise.

And if your package.json looks like this:

"dependencies": {
    "@angular/animations": "^4.0.0",
    "@angular/common": "^4.0.0",
    "@angular/compiler": "^4.0.0",
    "@angular/core": "^4.0.0",
    "@angular/forms": "^4.0.0",
    "@angular/http": "^4.0.0",
    "@angular/platform-browser": "^4.0.0",
    "@angular/platform-browser-dynamic": "^4.0.0"
}

... and all your node_modules/@angular/ packages have versions 5.x.x - all of them will be removed by npm. Furthermore - npm will remove all other packages, that are bound with /@angular/ packages. Thus, if node_modules folder has ~800 subfolders, when you replace package.json with lower versions, and simply install, for example, bootstrap (npm install bootstrap) - there will be ~150 subfolders in node_modules folder, already. Yep, minus 650.

It was like that couple days ago with npm v5.6.0 ... and yesterday they released version 6.0.0, and it still works in this not very smart order.

So, be careful, folks! 😎

Answer 64 · 2018-05-24T21:50:50.000Z

git+ssh dependencies are still removed in v6.1.0....

Missing dependencies after running `npm install` a second time

I'm opening this issue because:

What's going wrong?

How can the CLI team reproduce the problem?

supporting information:

Problems arise when only one artifact is present

First problem: missing `package-lock.json` results in `version` values becoming URLs

Major problem: missing `node_modules` and dependencies of devDependencies with a version spec of `latest`

`_requiredBy` becomes empty

Conclusion

I'm opening this issue because:

What's going wrong?

How can the CLI team reproduce the problem?

supporting information:

Problems arise when only one artifact is present

First problem: missing package-lock.json results in version values becoming URLs

Major problem: missing node_modules and dependencies of devDependencies with a version spec of latest

_requiredBy becomes empty

Conclusion

First problem: missing `package-lock.json` results in `version` values becoming URLs

Major problem: missing `node_modules` and dependencies of devDependencies with a version spec of `latest`

`_requiredBy` becomes empty