pinojs/pino

Memory Leak Pino, Pino Http, & Pino Pretty

Closed this issue · 11 comments

masb0ymas commented

I'm getting memory leak issues when running pino target pino-pretty, but when I run pino-pretty via command it's fine, with command like this :

node ./dist/main.js | pino-pretty

(node:33295) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 exit listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(Use node --trace-warnings ... to show where the warning was created)

export const logger = pino({
  level: env.NODE_ENV === 'production' ? 'info' : 'debug',
  // BUG Memory Leak
  transport: { target: 'pino-pretty', options: { colorize: true } },
})
mcollina commented

Thanks for reporting!

Can you provide steps to reproduce? We often need a reproducible example, e.g. some code that allows someone else to recreate your problem by just copying and pasting it. If it involves more than a couple of different file, create a new repository on GitHub and add a link to that.

masb0ymas commented

The case is like this: I have 3 libraries that use pino (with the transport target pino-pretty), then the boilerplate I created imports those libraries and the boilerplate also sets the pino configuration (with the transport target pino-pretty) this is where the problem lies.

so now my library code is fixed, just import pino without (transport pino-pretty), pino-pretty is only used in boilerplate (to reduce memory leaks)

mcollina commented

Can you create a repository that shows this behavior. None of us has been able to reproduce, there is something that you do in your boilerplate that is causing this.

masb0ymas commented

List repository :

how to improve my code to avoid memory leaks ?
Thanks.

mcollina commented

What are the steps to reproduce the problem with the above?

masb0ymas commented

The problem occur when i used transform configuration and not use the level configuration like :

export const logger = pino(
  {
    level: env.NODE_ENV === 'production' ? 'info' : 'debug',
    transport: {
      target: 'pino-pretty',
      options: {
        colorize: true,
      },
    },
  },
  pino.destination(`./logs/pino-${formatDate(new Date())}.log`)
)

it's work when :

export const logger = pino({
  level: env.NODE_ENV === 'production' ? 'info' : 'debug',
  // transport: { target: 'pino-pretty', options: { colorize: true } },
})

to pretty console with CLI:

node ./dist/main.js | pino-pretty
mcollina commented

I'm dumb and I don't have much time to figure out your project setup, could you please list it up for me?

  1. do x
  2. do y
  3. kaboom

Thanks

masb0ymas commented
  1. git clone https://github.com/masb0ymas/expresso-sequelize my-backend
  2. cd my-backend
  3. pnpm install
  4. copy .env.example to .env
  5. adjust .env
SEQUELIZE_CONNECTION=postgres
SEQUELIZE_HOST=127.0.0.1
SEQUELIZE_PORT=5432
SEQUELIZE_DATABASE=dev_expresso
SEQUELIZE_USERNAME=postgres
SEQUELIZE_PASSWORD=postgres
SEQUELIZE_SYNC=false
SEQUELIZE_LOGGING=true
SEQUELIZE_TIMEZONE=Asia/Jakarta
REDIS_HOST=127.0.0.1
REDIS_PORT=6379
REDIS_PASSWORD=
  1. yarn build
  2. yarn db:reset
  3. yarn start

You will see the CLI message

(node:79345) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 exit listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(Use `node --trace-warnings ...` to show where the warning was created)
jsumners commented

A minimal reproducible example should be the bare minimum code needed to trigger the issue, and easily runnable without any changes or extra code. What you are suggesting requires two database servers and many steps to run.

mcollina commented

The problem is somewhere in expresso or some related modules that is instantiating quite a few pino instances, causing the problem you are referring to:

buildStream Error
    at buildStream (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/lib/transport.js:21:30)
    at transport (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/lib/transport.js:115:10)
    at normalizeArgs (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/lib/tools.js:316:16)
    at pino (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/pino.js:87:28)
    at Object.<anonymous> (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/expresso-core@0.9.0/node_modules/expresso-core/lib/fs/createDir.js:11:35)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
buildStream Error
    at buildStream (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/lib/transport.js:21:30)
    at transport (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/lib/transport.js:115:10)
    at normalizeArgs (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/lib/tools.js:316:16)
    at pino (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/pino.js:87:28)
    at Object.<anonymous> (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/expresso-core@0.9.0/node_modules/expresso-core/lib/fs/deleteFile.js:11:35)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
buildStream Error
    at buildStream (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/lib/transport.js:21:30)
    at transport (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/lib/transport.js:115:10)
    at normalizeArgs (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/lib/tools.js:316:16)
    at pino (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/pino.js:87:28)
    at Object.<anonymous> (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/expresso-core@0.9.0/node_modules/expresso-core/lib/fs/readHtml.js:33:35)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
buildStream Error
    at buildStream (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/lib/transport.js:21:30)
    at transport (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/lib/transport.js:115:10)
    at normalizeArgs (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/lib/tools.js:316:16)
    at pino (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/pino.js:87:28)
    at Object.<anonymous> (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/expresso-core@0.9.0/node_modules/expresso-core/lib/fs/writeFile.js:33:35)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
buildStream Error
    at buildStream (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/lib/transport.js:21:30)
    at transport (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/lib/transport.js:115:10)
    at normalizeArgs (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/lib/tools.js:316:16)
    at pino (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/pino.js:87:28)
    at Object.<anonymous> (/Users/matteo/tmp/expresso-sequelize/dist/config/pino.js:30:31)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
buildStream Error
    at buildStream (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/lib/transport.js:21:30)
    at transport (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/lib/transport.js:115:10)
    at normalizeArgs (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/lib/tools.js:316:16)
    at pino (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/pino.js:87:28)
    at Object.<anonymous> (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/expresso-provider@0.9.0/node_modules/expresso-provider/lib/smtp.js:13:35)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
    at require (node:internal/modules/cjs/helpers:121:18)
    at Object.<anonymous> (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/expresso-provider@0.9.0/node_modules/expresso-provider/lib/index.js:6:14)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
buildStream Error
    at buildStream (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/lib/transport.js:21:30)
    at transport (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/lib/transport.js:115:10)
    at normalizeArgs (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/lib/tools.js:316:16)
    at pino (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/pino.js:87:28)
    at Object.<anonymous> (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/expresso-provider@0.9.0/node_modules/expresso-provider/lib/storage.js:40:35)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
    at require (node:internal/modules/cjs/helpers:121:18)
    at Object.<anonymous> (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/expresso-provider@0.9.0/node_modules/expresso-provider/lib/index.js:10:17)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
buildStream Error
    at buildStream (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/lib/transport.js:21:30)
    at transport (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/lib/transport.js:115:10)
    at normalizeArgs (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/lib/tools.js:316:16)
    at pino (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/pino.js:87:28)
    at Object.<anonymous> (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/expresso-hooks@0.9.0/node_modules/expresso-hooks/lib/multer/useMulter.js:13:35)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
    at require (node:internal/modules/cjs/helpers:121:18)
    at Object.<anonymous> (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/expresso-hooks@0.9.0/node_modules/expresso-hooks/lib/index.js:17:14)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
    at require (node:internal/modules/cjs/helpers:121:18)
    at Object.<anonymous> (/Users/matteo/tmp/expresso-sequelize/dist/app/middleware/authorization.js:12:24)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
buildStream Error
    at buildStream (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/lib/transport.js:21:30)
    at transport (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/lib/transport.js:115:10)
    at normalizeArgs (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/lib/tools.js:316:16)
    at pino (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/pino@8.15.3/node_modules/pino/pino.js:87:28)
    at Object.<anonymous> (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/expresso-hooks@0.9.0/node_modules/expresso-hooks/lib/token/useToken.js:35:35)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
    at require (node:internal/modules/cjs/helpers:121:18)
    at Object.<anonymous> (/Users/matteo/tmp/expresso-sequelize/node_modules/.pnpm/expresso-hooks@0.9.0/node_modules/expresso-hooks/lib/index.js:18:14)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
    at require (node:internal/modules/cjs/helpers:121:18)
    at Object.<anonymous> (/Users/matteo/tmp/expresso-sequelize/dist/app/middleware/authorization.js:12:24)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)

I added a console.log(new Error().stack) in

pino/lib/transport.js

Line 20 in c07cd48

function buildStream (filename, workerData, workerOpts) {

It seems you are instantiating pino instances everywhere, therefore causing this problem.

github-actions commented

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.