fluent-plugin-file-alternative

File output plugin alternative implementation, is 100% compatible with fluentd built-in 'out_file', and added many options to format output as you want.

FileAlternativeOutput slices data by time (for specified units), and store these data as plain text on hdfs. You can specify to:

format whole data as serialized JSON, single attribute or separated multi attributes
include time as line header, or not
include tag as line header, or not
change field separator (default: TAB)
add new line as termination, or not

And you can specify output file path as:

Standard out_file way
- configure 'path /path/to/dir/access'
- and 'time_slice_format %Y%m%d'
- got '/path/to/dir/access.20120316.log'
Alternative style
- configure 'path /path/to/dir/access.%Y%m%d.log' only
- got '/path/to/dir/access.20120316.log'

And, gzip compression is also supported.

Configuration

FileAlternativeOutput

Standard out_file way (hourly log, compression, time-tag-json):

<match out.**>
  type file\_alternative
  path /var/log/service/access.*.log
  time\_slice\_output %Y%m%d_%H
  compress gzip
</match>

By this configuration, in gzip compressed file '/var/log/service/access.20120316_23.log.gz', you get:

2012-03-16T23:59:40 [TAB] out.service.xxx [TAB] {"field1":"value1","field2":"value2"}
2012-03-16T23:59:40 [TAB] out.service.xxx [TAB] {"field1":"value1","field2":"value2"}
2012-03-16T23:59:40 [TAB] out.service.xxx [TAB] {"field1":"value1","field2":"value2"}
2012-03-16T23:59:40 [TAB] out.service.xxx [TAB] {"field1":"value1","field2":"value2"}

If you don't want fluentd-time and tag in written file, and messages with single attribute (as raw full apache log with newline):

<match out.**>
  type file\_alternative
  path /var/log/service/access.%Y%m%d_%H.log
  compress gzip
  output_include_time false
  output_include_tag false
  output_data_type attr:message
  add_newline false
</match>

Then, you will get:

192.168.0.1 - - [16/Mar/2012:23:59:40 +0900] "GET /content/x HTTP/1.1" 200 -
192.168.0.1 - - [16/Mar/2012:23:59:40 +0900] "GET /content/x HTTP/1.1" 200 -
192.168.0.1 - - [16/Mar/2012:23:59:40 +0900] "GET /content/x HTTP/1.1" 200 -

TODO

consider what to do next
patches welcome!

muddydixon/fluent-plugin-file-alternative

fluent-plugin-file-alternative

Configuration

FileAlternativeOutput

TODO

Copyright