Using a transport to transform logs and then write to a custom stream

[justindeocampodd]

Hello,

I am currently trying to change Pino's base logging structure using a custom transport pipeline but I am having some trouble piping the log data to a custom stream if one is provided when I initialize the logger.

Here is what my transport pipeline looks like:

const exampleTransformer = async (options: any): Promise<Transform> => {
  return build(
    function (source: any) {
      const transportStream = new Transform({
        autoDestroy: true,
        objectMode: true,
        transform(chunk: any, enc: BufferEncoding, cb: TransformCallback) {
          const logObj = transformLogObject(chunk) // this is just a function that removes some properties of the logs
          this.push(`${JSON.stringify(logObj)}\n`)
          cb()
        }
      })
      console.log(options)

     // THIS IS WHERE IM CONFUSED
      if (options && options.customStream) {
        pipeline(source, transportStream, options.customStream, () => {});
      } else {
        console.log("DEFAULT STREAM")
        pipeline(source, transportStream, () => {});
      }
      return transportStream
    },
    {
      // This is needed to be able to pipeline transports.
      enablePipelining: true
    }
  )
}
export = exampleTransformer

When I initialize my logger I am using this as my transport pipeline

transport: {
        pipeline: [
          {
            target: './example-transformer.js',
            options: {
              customStream: stream // this is the custom stream that I am trying to pass into my transport pipeline
            }
          },
          {
            target: 'pino/file'
          }]
      },

The problem is the transport pipeline is only seeing the custom stream that I am passing in as undefined. I believe that the issue is that Pino uses a worker thread to asynchronously do the transport work which means that the custom stream I am passing in when initializing the logger isn't guaranteed to exist when getting passed into it.

My questions are:

1. Is it possible to use transport pipelines to write to a custom stream that I pass in when initializing the logger?
2. Is it possible to unit test the output of what happens after a transport pipeline? I'm finding it difficult because of the inability to write to a custom stream and the asynchronous nature of transport pipelines
3. What is the best practice for reshaping the Pino logs with full customization (e.g. removing the 'msg' variable and being able to nest it inside of some other property in the logger). I feel like transports gave me full customization over how I want the log to look, but it gives severe limitations in where I can write the output to, making it difficult to test

Thanks for the help!

[mcollina]

Is it possible to use transport pipelines to write to a custom stream that I pass in when initializing the logger?

No. The transport are run in a separate thread, and functions cannot be transferred.

Is it possible to unit test the output of what happens after a transport pipeline? I'm finding it difficult because of the inability to write to a custom stream and the asynchronous nature of transport pipelines

Write a unit test for your transport directly. There are a few examples on how to deal with this inside this repo, however this is mostly not needed: any bug in that logic should be done here.

What is the best practice for reshaping the Pino logs with full customization (e.g. removing the 'msg' variable and being able to nest it inside of some other property in the logger). I feel like transports gave me full customization over how I want the log to look, but it gives severe limitations in where I can write the output to, making it difficult to test

Use transports. Note that you can code your transform so that it is just a node stream, (see pino-pretty), so that pino(buildYourTransport()) is possible, allowing for your testing.

[justindeocampodd]

Write a unit test for your transport directly. There are a few examples on how to deal with this inside this repo, however this is mostly not needed: any bug in that logic should be done here.

Ahh ok this makes sense. I couldn't find the examples but I know what you mean.

Use transports. Note that you can code your transform so that it is just a node stream, (see pino-pretty), so that pino(buildYourTransport()) is possible, allowing for your testing.

Do you think you could explain this a bit more? Not sure I'm understanding what is meant by pino(buildYourTransport()). Not seeing how this gets around the issue of being able to write to a stream that is passed in on initialization. I know you said in the first answer that bc transports are run in a worker thread its not possible, but this seems like a huge limitation of transports doesn't it? Essentially what you're saying is if I use a transport I have to write to std.out?

const opts: pino.LoggerOptions = {
   transport: {
        pipeline: [
          {
            target: './example-transformer.js'
          },
          {
            target: 'pino/file'
          }
        ]
      },
}

This is currently what my transport pipeline looks like. From the docs it seems pino/file writes to std.out by default, and you can pass in a destination as an option, but the destination is just a path to a file and not a custom stream like I'd like to do.

[mcollina]

This is currently what my transport pipeline looks like. From the docs it seems pino/file writes to std.out by default, and you can pass in a destination as an option, but the destination is just a path to a file and not a custom stream like I'd like to do.

Due to JavaScript limitations, you cannot pass a custom stream: functions cannot pass through threads, only serializable objects. I already said this in #1786 (comment).

You'll have to construct your custom stream as a transform itself:

const opts: pino.LoggerOptions = {
   transport: {
        pipeline: [
          {
            target: './example-transformer.js'
          },
          {
            target: './example-destination.js',
            options: { ... }
          }
        ]
      },
}

Essentially what you're saying is if I use a transport I have to write to std.out?

No, you can write wherever you want or need, there are plenty of transports available.

Do you think you could explain this a bit more? Not sure I'm understanding what is meant by pino(buildYourTransport()).

You want to have implement this same API https://github.com/pinojs/pino-pretty/blob/fa386a964cce7d7c5cbeb4ce5bb8d28785001acd/test/basic.test.js#L1013-L1038, which you do by using the pino-abstract-transport (which you already seem to be using).
Take a look at https://github.com/pinojs/pino-pretty/blob/fa386a964cce7d7c5cbeb4ce5bb8d28785001acd/index.js#L212-L246 as well.