How to rotate logs in a separate thread

[noweh]

Hello,

I am trying to use pino for logging in to my node app Server and I do have some large logs coming, so rotating the files every day would be more practical.

So I used pino.multistream() and require('file-stream-rotator')

My code works, but for performance reasons, I would not like to use the streams in the main thread.

according to the doc, I should use pino.transport():

[pino.multistream()] differs from pino.transport() as all the streams will be executed within the main thread, i.e. the one that created the pino instance.
https://github.com/pinojs/pino/releases?page=2

However, I can't manage to combine pino.transport() and file-stream-rotator.

my code that does not work completely
-> logs the first entries, but is not exportable because it blocks the script with the error

throw new Error('the worker has exited')

Main file

const pino = require('pino')

const transport = pino.transport({
  target: './custom-transport.js'
})

const logger = pino(transport)
logger.level = 'info'

logger.info('Pino: Start Service Logging...')

module.exports = {
  logger
}

custom-transport.js file

const { once } = require('events')
const fileStreamRotator = require('file-stream-rotator')

const customTransport = async () => {
  const stream = fileStreamRotator.getStream({ filename: 'myfolder/custom-logger.log', frequency: 'daily' })
  await once(stream, 'open')
  return stream
}

module.exports = customTransport

Thanks !

[mcollina]

What do you mean by

but is not exportable

[mcollina]

By looking at https://github.com/rogerc/file-stream-rotator/blob/2b41e6a9a6447394999a4ea03a695d8398238969/FileStreamRotator.js#L540, it looks like this module does not return a stream but an event emitter.

I think part of the problem is that the file-stream-rotator is not emitting the 'open' event.

Another option is to add a PassThrough stream in between.

--

a better solution to this problem is to reimplement this using sonic-boom as it as all the capabilities to be a better rotator. That will require a new module.

[feugy]

Hello folks.
I'd like to volunteer on this one.

I did a quick spike with sonic boom, and it's indeed very easy to create a transport that rotates files based on their size.

@noweh I will not aim at supporting all the options file-stream-rotator has. Can you tell me what you'll need most:

1. rotate based on daily/test frequency
2. rotate based on custom frequency
3. rotate based on file size
4. remove old logs based on a maximum number of files

Thanks

[mcollina]

Go for it!

[feugy]

Another silly question @noweh. When rotating files, there are multiple strategies:

1. current log file name is the previous file name +1(in particular, when using max size):

   log.1 (older)
   log.2  (current)
   

2. current log file name is constant, and older logs are shifted by one on rotation:

   log (current)
   log.1 (older)
   log.2 (odest)
   

I believe file-stream-rotator implements the first, but Unix logrotate command implements the second.

A strong preference?

Note: this may also apply when rotating on frequency. The date will be part of the log file name, but we may combine both date and size to control rotation.

[mcollina]

I would use https://github.com/pinojs/sonic-boom#sonicboomreopenfile to open the new file. This will make sure there are no lost log lines.

[noweh]

Thanks for the update. Rotate based on daily would be great. Without preference for the files name: the simplest is the best 👍😃

[feugy]

It's not easy to get it reviewed, but here is a "draft" implementation.

https://github.com/feugy/pino-roll

It needs more automated testing but may be just enough for @noweh to do some testing.
Let me know what you think, and I'll resume this next week.

[noweh]

Thanks again! I hope to find time at the beginning of next week to test all this.

[mcollina]

Could you add it to per known trabsports in the docs @feugy ?

[feugy]

@mcollina, yes I will, when we'll be happy with the proposal. Let's see how useful it is first.

[feugy]

Hello gents!

@noweh, @DRoet I've published pino-roll@1.0.0-rc.1, please have a try and let me know.

@mcollina I'll add this to the list of known transport afterward

[Fdawgs]

@noweh Did you manage to resolve this?

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.