Fluentd Filter plugin to concatenate multiline log separated in multiple events.
fluent-plugin-concat | fluentd | ruby |
---|---|---|
>= 2.0.0 | >= v0.14.0 | >= 2.1 |
< 2.0.0 | >= v0.12.0 | >= 1.9 |
Add this line to your application's Gemfile:
gem 'fluent-plugin-concat'
And then execute:
$ bundle
Or install it yourself as:
$ gem install fluent-plugin-concat
key (string) (required)
The key for part of multiline log.
separator (string) (optional)
The separator of lines.
Default value is "\n"
.
n_lines (integer) (optional)
The number of lines.
This is exclusive with multiline_start_regex
.
multiline_start_regexp (string) (optional)
The regexp to match beginning of multiline.
This is exclusive with n_lines.
multiline_end_regexp (string) (optional)
The regexp to match ending of multiline.
This is exclusive with n_lines.
continuous_line_regexp (string) (optional)
The regexp to match continuous lines.
This is exclusive with n_lines.
stream_identity_key (string) (optional)
The key to determine which stream an event belongs to.
flush_interval (integer) (optional)
The number of seconds after which the last received event log will be flushed. If specified 0, wait for next line forever.
timeout_label (string) (optional)
The label name to handle events caused by timeout.
use_first_timestamp (bool) (optional)
Use timestamp of first record when buffer is flushed.
Default value is false
.
partial_key (string) (optional)
The field name that is the reference to concatenate records
partial_value (string) (optional)
The value stored in the field specified by partial_key that represent partial log
keep_partial_key (bool) (optional)
If true, keep partial_key in concatenated records
Default value is false
.
use_partial_metadata (bool) (optional)
Use partial metadata to concatenate multiple records
keep_partial_metadata (bool) (optional)
If true, keep partial metadata
Every 10 events will be concatenated into one event.
<filter docker.log>
@type concat
key message
n_lines 10
</filter>
Specify first line of multiline by regular expression.
<filter docker.log>
@type concat
key message
multiline_start_regexp /^Start/
</filter>
You can handle timeout events and remaining buffers on shutdown this plugin.
<label @ERROR>
<match docker.log>
@type file
path /path/to/error.log
</match>
</label>
Handle timeout log lines the same as normal logs.
<filter **>
@type concat
key message
multiline_start_regexp /^Start/
flush_interval 5
timeout_label @NORMAL
</filter>
<match **>
@type relabel
@label @NORMAL
</match>
<label @NORMAL>
<match **>
@type stdout
</match>
</label>
Handle single line JSON from Docker containers.
<filter **>
@type concat
key message
multiline_end_regexp /\n$/
</filter>
Handle Docker logs splitted in several parts (using partial_message
), and do not add new line between parts.
<filter>
@type concat
key log
partial_key partial_message
partial_value true
separator ""
</filter>
(Docker v19.03+) Handle Docker logs splitted in several parts (using use_partial_metadata
), and do not add new line between parts.
<filter>
@type concat
key log
use_partial_metadata true
separator ""
</filter>
Handle Docker logs splitted in several parts (using newline detection), and do not add new line between parts (prior to Docker 18.06).
<filter **>
@type concat
key log
multiline_end_regexp /\\n$/
separator ""
</filter>
Handle containerd/cri in Kubernetes.
<source>
@type tail
path /var/log/containers/*.log
<parse>
@type regexp
expression /^(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+Z) (?<output>\w+) (?<partial_flag>[FP]) (?<message>.+)$/
</parse>
tag k8s
@label @CONCAT
</source>
<label @CONCAT>
<filter k8s>
@type concat
key message
partial_key partial_flag
partial_value P
</filter>
<match k8s>
@type relabel
@label @OUTPUT
</match>
</label>
<label @OUTPUT>
<match>
@type stdout
</match>
</label>
- Fork it
- Create your feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Add some feature'
) - Push to the branch (
git push origin my-new-feature
) - Create new Pull Request
The gem is available as open source under the terms of the MIT License.