Any way to return "msg" (in prod) as JSON?

[armenr]

Hey @niftylettuce!

We're using DataDog, and it's friendly to "natural JSON" a la pino.

I've read the documentation and understand that whatever is returned @ the msg property has to be a string, and that it complies with the apache-friendly log RFCs, because you're a man of taste and experience.

BUT, is there any way to get JSON back? Or maybe have the choice between apache-friendly and JSON?

Thanks for all your work, man!
_Armen

[niftylettuce]

Can you give me an example / code snippet of what you're trying to do and the expected vs. actual output? I would definitely accept a PR for sure man or fix this myself for you. Just need to know specifically what you mean

[armenr]

Thanks dude!

Example prod log entry sent to DataDog:

{
   "level":30,
   "time":1573656121209,
   "pid":12011,
   "hostname":"ip-10-0-12-50",
   "msg":"0.0.0.0 - - 13/Nov/2019:14:42:01 +0000 \"GET /static/favicon.ico HTTP/1.1\" 200 - {\"id\":\"5dcc163992279d2eeb055564\",\"timestamp\":\"2019-11-13T14:42:01.000Z\",\"request\":{\"method\":\"GET\",\"query\":{},\"headers\":{\"accept\":\"image/webp,image/apng,image/*,*/*;q=0.8\",\"accept-encoding\":\"gzip, deflate\",\"accept-language\":\"en-US,en;q=0.9\",\"cache-control\":\"no-cache\",\"connection\":\"keep-alive\",\"cookie\":\"dtCookie=v_4_srv_1_sn_83AA82EA4B50C9EE45C851779D892DC8_perc_100000_ol_0_mul_1\",\"host\":\"1.1.1.1:3000\",\"pragma\":\"no-cache\",\"referer\":\"http://1.1.1.1:3000/\",\"user-agent\":\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36\",\"x-dynatrace-application\":\"1%3B1%3Bea7c4b59f27d43eb\",\"x-dynatrace-origin-url\":\"http://1.1.1.1:3000/static/favicon.ico\",\"username\":\"NoAuth\"},\"cookies\":{\"dtCookie\":\"v_4_srv_1_sn_83AA82EA4B50C9EE45C851779D892DC8_perc_100000_ol_0_mul_1\"},\"url\":\"/static/favicon.ico\",\"timestamp\":\"2019-11-13T14:42:01.205Z\",\"id\":\"5d5ac094-9cfb-4789-abb6-cfc8178f85ad\",\"http_version\":\"1.1\"},\"user\":{\"ip_address\":\"0.0.0.0\"},\"response\":{\"headers\":{\"x-powered-by\":\"Good Vibes 1.0\",\"access-control-allow-origin\":\"*\",\"surrogate-control\":\"no-store\",\"cache-control\":\"no-store, no-cache, must-revalidate, proxy-revalidate\",\"pragma\":\"no-cache\",\"expires\":\"0\",\"x-content-type-options\":\"nosniff\",\"x-frame-options\":\"SAMEORIGIN\",\"x-xss-protection\":\"1; mode=block\",\"x-request-id\":\"5d5ac094-9cfb-4789-abb6-cfc8178f85ad\",\"x-request-username\":\"NoAuth\",\"accept-ranges\":\"bytes\",\"last-modified\":\"Mon, 11 Nov 2019 21:07:42 GMT\",\"etag\":\"W/\\\"1a47-16e5c4b3130\\\"\",\"content-type\":\"image/x-icon\",\"x-response-time\":\"1.256ms\",\"vary\":\"Accept-Encoding\",\"content-encoding\":\"gzip\",\"date\":\"Wed, 13 Nov 2019 14:42:01 GMT\",\"connection\":\"keep-alive\",\"transfer-encoding\":\"chunked\"},\"http_version\":\"1.1\",\"status_code\":200,\"reason_phrase\":\"OK\",\"timestamp\":\"2019-11-13T14:42:01.000Z\",\"duration\":1.256},\"duration\":0.534569,\"app\":{\"node\":\"v12.13.0\",\"environment\":\"production\",\"hostname\":\"ip-10-0-12-50\",\"pid\":12011}}",
   "v":1
}

Pino + CabinJS thangz:

const pinoProd = pino({
  customLevels: {
    log: 30,
  },
});

const pinoDev = pino({
  prettyPrint: {
    colorize: true,
  },
});

const cabin = new Cabin({
  capture: false,
  axe: {
    logger: NODE_ENV === 'production' ? pinoProd : pinoDev,
  },
});

Desired behavior:

In the log output, you'll notice that everything at the "msg" value is stringified JSON.

I'm wondering if there's a way to return all of this - in prod - as pure and total JSON, since tools like DataDog and GrayLog are friendly to JSON, and would automatically detect and extract the key/value pairs as labels/fields.

I know it's absolutely ANTITHETICAL to the concept of the apache log standards, but...I'm shamelessly asking :)

[niftylettuce]

Have you tried specifying a custom message function? Not sure if that would fix it for you - let me know otherwise.

https://github.com/cabinjs/cabin#options
https://github.com/cabinjs/cabin/blob/master/src/message.js

[armenr]

I've read through that - not to be a nag or a baby - but I'm just trying to figure out (maybe with an example) what a custom function would look like?

I'm actually more confused now, since the code I see is this:

function apacheCommonLogFormat(options) {
  const { req, res, ctx } = options;

  const creds = auth(req);

  const startTime = getStartTime(req);

  return `${ctx ? ctx.ip : req.ip} - ${creds ? creds.name : '-'} ${clfDate(
    startTime
  )} "${req.method} ${req.url} HTTP/${req.httpVersionMajor}.${
    req.httpVersionMinor
  }" ${res.statusCode} ${res.getHeader('content-length') || '-'}`;
}

but it looks like I'm getting WAY more than that in my logs to DataDog...sorry for sounding illiterate.

[niftylettuce]

That code returns a string as the message. If you want an object, you could do something like:

function message(options) {
  const { req, res, ctx } = options;
  const creds = auth(req);
  const startTime = getStartTime(req);

  return {
    ip: ctx ? ctx.ip : req.ip,
    creds: creds ? creds.name : '-',
    clf_date: clfDate(startTime),
    method: req.method,
    url: req.url,
    http_version: req.httpVersionMajor + '.' + req.httpVersionMinor,
    status_code: res.statusCode,
    content_length: res.getHeader('content-length') || '-'
  };
}

// ...

const cabin = new Cabin({
  capture: false,
  axe: {
    logger: NODE_ENV === 'production' ? pinoProd : pinoDev,
  },
  message // <--
});

[niftylettuce]

Pino itself might be stringifying the message, I am unsure.