logstash-plugins/logstash-filter-mutate

rename fields after useragent plugin

moix opened this issue · 0 comments

moix commented

Hello,
I'm applying a pipeline for an apm setup where I'm using two filters, geoip and useragent at the moment.

input {
  beats {
    port => 5044
  }
}

filter {
  geoip {
    source => [ "[context][request][socket][remote_address]" ]
    target => "user.geoip"
  }
  useragent {
    source => [ "[user_agent][original]" ]
    target => "user_agent.fields"
  }
}

So I got something like following for the user agent for example:

    "user_agent": {
      "original": "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:64.0) Gecko/20100101 Firefox/64.0"
    },
. . .
    "user_agent.fields": {
      "minor": "0",
      "major": "64",
      "os": "Fedora",
      "build": "",
      "name": "Firefox",
      "os_name": "Fedora",
      "device": "Other"
    },

which is fine, but the index definition of apm is slightly different:

      "user_agent": {
        "dynamic": false,
        "properties": {
          "device": {
            "properties": {
              "name": {
                "ignore_above": 1024,
                "type": "keyword"
              }
            }
          },
          "name": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "original": {
            "fields": {
              "text": {
                "norms": false,
                "type": "text"
              }
            },
            "ignore_above": 1024,
            "type": "keyword"
          },
          "os": {
            "properties": {
              "family": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "full": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "kernel": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "name": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "platform": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "version": {
                "ignore_above": 1024,
                "type": "keyword"
              }
            }
          },
          "version": {
            "ignore_above": 1024,
            "type": "keyword"
          }
        }

so I thought on performing a move, for example useragent os_name should be renamed with:

  mutate {
    rename => { "[user_agent][fields][os_name]" => "[user_agent][os][name]" }
  }

however this is not working. I'm guessing that filters are not applied in order maybe, son when mutate tries to rename, the user_agent hasn't run yet?

thanks