/fluent-plugin-map

fluent-plugin-map is the non-buffered plugin that can convert an event log to different event log(s)

Primary LanguageRubyApache License 2.0Apache-2.0

fluent-plugin-map

Build Status

fluent-plugin-map(out_map) is the non-buffered plugin that can convert an event log to different event log(s)

Requirements

fluent-plugin-map fluentd ruby
>= 0.2.0 >= v0.14.0 >= 2.1
< 0.2.0 >= v0.12.0 >= 1.9

MapFilter

Configuration

parameter description default
map Specify rule to convert an event log nil
time Specify rule to convert a time. The format of time is an integer number of seconds since the Epoch nil
record Specify rule to convert a record.The format of record is hash. nil
multi Enable to output multi logs false
timeout Specify timeout 1
format Specify format. If map is specified, the format is map. If time and record is specified, the format is record. (map, record, multimap) nil

Example

This sample config filter code file and time file.

<source>
  @type tail
  format apache
  path /var/log/httpd-access.log
  tag tag
  @label @raw
</source>
<label @raw>
  <match **>
    @type copy
    <store>
      @type relabel
      @label @code
    </store>
    <store>
      @type relabel
      @label @time
    </store>
  </match>
</label>
<label @code>
  <filter **>
    @type map
    map ([time, {"code" => record["code"].to_i}])
  </filter>
  <match **>
    @type file
    path code.log
  </match>
</label>
<label @time>
  <filter **>
    @type map
    map ([time, {"time" => record["time"].to_i}])
  </filter>
  <match **>
    @type file
    path time.log
  </match>
</label>

The parameter "map" can use 2 variables in event log; time, record. The format of time is an integer number of seconds since the Epoch. The format of record is hash. The config file parses # as the begin of comment. So the "map" value cannot use #{tag} operation. This plugin can output multi logs by seting multi to true.

If you don't use multi option, you can use time, record parameter. The 2 following filter directive is same:

<filter tag>
  @type map
  map ([time, {"code" => record["code"].to_i}])
</filter>
<filter tag>
  @type map
  time time
  record ({"code" => record["code"].to_i})
</filter>

MapOutput

Configuration

parameter description default
map Specify rule to convert an event log nil
time Specify rule to convert a time. The format of time is an integer number of seconds since the Epoch nil
record Specify rule to convert a record.The format of record is hash. nil
multi Enable to output multi logs false
timeout Specify timeout 1
format Specify format. If map is specified, the format is map. If time and record and (tag or key) are specified, the format is record. (map, record) nil
key Specify the key in record. This parameter is deprecated. nil
tag Specify the tag in record nil

Example

This sample config output code file and time file.

<source>
  @type tail
  format apache
  path /var/log/httpd-access.log
  tag tag
</source>
<match tag>
  @type map
  map ([["code." + tag, time, {"code" => record["code"].to_i}], ["time." + tag, time, {"time" => record["time"].to_i}]])
  multi true
</match>
<match code.tag>
  @type file
  path code.log
</match>
<match time.tag>
  @type file
  path time.log
</match>

The parameter "map" can use 3 variables in event log; tag, time, record. The format of time is an integer number of seconds since the Epoch. The format of record is hash. The config file parses # as the begin of comment. So the "map" value cannot use #{tag} operation. This plugin can output multi logs by seting multi to true.

If you don't use multi option, you can use key, time, record parameter. The 2 following match directive is same:

<match tag>
  @type map
  map (["code." + tag, time, {"code" => record["code"].to_i}])
</match>
<match tag>
  @type map
  tag ("code." + tag)
  time time
  record ({"code" => record["code"].to_i})
</match>

Note

you have to wrap some configuration values with parenthesis like ("code." + tag), to avoid parsing by Fluentd itself. See also: Config File Syntax - Fluentd

Copyright

  • Copyright (c) 2015- Tomita Kohei
  • Apache License, Version 2.0