verekia/js-stack-from-scratch

PM2 causing runaway process deletion resulting in major system crash and file corruption (El Capitan 10.11.6)

Closed this issue · 2 comments

Aerlinger commented

I'm having a rather serious issue with this step in the tutorial: https://github.com/verekia/js-stack-from-scratch/blob/master/tutorial/03-express-nodemon-pm2.md#pm2 where my system crashes in a very peculiar way after running yarn prod:start followed yarn prod:stop. Specifically, pm2 delete all seems to ravage my process list resulting in a unstable system with possible side effects in the filesystem.

I'm running OSX El Capitan 10.11.6. PM2 version is 2.4.2 on Node 7.4.4.

My Steps:

  1. git checkout https://github.com/verekia/js-stack-walkthrough
  2. cd to this directory: https://github.com/verekia/js-stack-walkthrough/tree/master/03-express-nodemon-pm2
  3. yarn install
  4. yarn prod:build
  5. yarn prod:start (note: nothing is running on port 8000 at this point)

Output

yarn prod:start v0.21.3
$ cross-env NODE_ENV=production pm2 start lib/server && pm2 logs
[PM2] Spawning PM2 daemon with pm2_home=/Users/Aerlinger/.pm2
[PM2] PM2 Successfully daemonized
[PM2] Starting /Users/Aerlinger/Github/js-stack-from-scratch/js-stack-walkthrough/03-express-nodemon-pm2/lib/server in fork_mode (1 instance)
[PM2] Done.
┌──────────┬────┬──────┬──────┬────────┬─────────┬────────┬─────┬───────────┬──────────┐
│ App name │ id │ mode │ pid  │ status │ restart │ uptime │ cpu │ mem       │ watching │
├──────────┼────┼──────┼──────┼────────┼─────────┼────────┼─────┼───────────┼──────────┤
│ server   │ 0  │ fork │ 3283 │ online │ 0       │ 0s     │ 0%  │ 13.6 MB   │ disabled │
└──────────┴────┴──────┴──────┴────────┴─────────┴────────┴─────┴───────────┴──────────┘
 Use `pm2 show <id|name>` to get more details about an app
[TAILING] Tailing last 15 lines for [all] processes (change the value with --lines option)
/Users/Aerlinger/.pm2/pm2.log last 15 lines:
PM2        | 2017-03-07 09:05:18: Time                 : Tue Mar 07 2017 09:05:18 GMT-0500 (EST)
PM2        | 2017-03-07 09:05:18: PM2 version          : 2.4.2
PM2        | 2017-03-07 09:05:18: Node.js version      : 7.4.0
PM2        | 2017-03-07 09:05:18: Current arch         : x64
PM2        | 2017-03-07 09:05:18: PM2 home             : /Users/Aerlinger/.pm2
PM2        | 2017-03-07 09:05:18: PM2 PID file         : /Users/Aerlinger/.pm2/pm2.pid
PM2        | 2017-03-07 09:05:18: RPC socket file      : /Users/Aerlinger/.pm2/rpc.sock
PM2        | 2017-03-07 09:05:18: BUS socket file      : /Users/Aerlinger/.pm2/pub.sock
PM2        | 2017-03-07 09:05:18: Application log path : /Users/Aerlinger/.pm2/logs
PM2        | 2017-03-07 09:05:18: Process dump file    : /Users/Aerlinger/.pm2/dump.pm2
PM2        | 2017-03-07 09:05:18: Concurrent actions   : 2
PM2        | 2017-03-07 09:05:18: SIGTERM timeout      : 1600
PM2        | 2017-03-07 09:05:18: ===============================================================================
PM2        | 2017-03-07 09:05:18: Starting execution sequence in -fork mode- for app name:server id:0
PM2        | 2017-03-07 09:05:18: App name:server id:0 online

/Users/Aerlinger/.pm2/logs/server-error-0.log last 15 lines:
0|server   |     at Function.listen (/Users/Aerlinger/Code/javascript/js-stack-from-scratch/node_modules/express/lib/application.js:618:24)
0|server   |     at Object.<anonymous> (/Users/Aerlinger/Code/javascript/js-stack-from-scratch/lib/server/index.js:31:5)
0|server   |     at Module._compile (module.js:571:32)
0|server   |     at Object.Module._extensions..js (module.js:580:10)
0|server   | Error: listen EADDRINUSE :::8000
0|server   |     at Object.exports._errnoException (util.js:1022:11)
0|server   |     at exports._exceptionWithHostPort (util.js:1045:20)
0|server   |     at Server._listen2 (net.js:1262:14)
0|server   |     at Server._listen2 (/Users/Aerlinger/Code/javascript/js-stack-from-scratch/node_modules/async-listener/index.js:51:23)
0|server   |     at listen (net.js:1298:10)
0|server   |     at Server.listen (net.js:1376:9)
0|server   |     at Function.listen (/Users/Aerlinger/Code/javascript/js-stack-from-scratch/node_modules/express/lib/application.js:618:24)
0|server   |     at Object.<anonymous> (/Users/Aerlinger/Code/javascript/js-stack-from-scratch/lib/server/index.js:31:5)
0|server   |     at Module._compile (module.js:571:32)
0|server   |     at Object.Module._extensions..js (module.js:580:10)

/Users/Aerlinger/.pm2/logs/server-out-0.log last 15 lines:
0|server   | Server running on port 8000 (production).
0|server   | Server running on port 8000 (production).
0|server   | Server running on port 8000 (production).
0|server   | Server running on port 8000 (production).
0|server   | Server running on port 8000 (production).

At this point running yarn prod:stop seems to "fork bomb" the system and terminate several running process. The result is a frozen system where the OS is still running, but nothing can be opened, interacted with and I can hardly reboot the machine. Also, some config files for running process such as those used by iTerm/zshell are possibly corrupted and I'm working to get them back.

Note that this was run directly from code provided here: https://github.com/verekia/js-stack-walkthrough/tree/master/03-express-nodemon-pm2

I realize that the core issue here is likely with PM2, specifically the pm2 delete all command, conflicting with some other process on my machine, but wanted to flag the issue here as it's rather serious and has caused me to lose work and may have had other side effects on my system.

icflorescu commented

Can't test this on macOS at the moment, but as a possible workaround, try to replace pm2 delete all with pm2 delete server and see if it works better. It should, since pm2 start lib/server creates an app named server...

verekia commented

@Aerlinger Thank you for pointing this out, and sorry about that. I'm updating the tutorial to use pm2 delete server, as @icflorescu suggested.