fluent-plugin-grepcounter

Fluentd plugin to count the number of matched messages, and emit if exeeds the threshold.

Configuration

Assume inputs from another plugin are as belows:

syslog.host1: {"message":"20.4.01/13T07:02:11.124202 INFO GET /ping" }
syslog.host1: {"message":"20.4.01/13T07:02:13.232645 WARN POST /auth" }
syslog.host1: {"message":"20.4.01/13T07:02:21.542145 WARN GET /favicon.ico" }
syslog.host1: {"message":"20.4.01/13T07:02:43.632145 WARN POST /login" }

An example of grepcounter configuration:

<match syslog.**>
  type grepcounter
  count_interval 60
  input_key message
  regexp WARN
  exclude favicon.ico
  threshold 1
  add_tag_prefix warn.count
</match>

Then, output bocomes as belows (indented):

warn.count.syslog.host1: {
  "count":2,
  "message":["20.4.01/13T07:02:13.232645 WARN POST /auth","20.4.01/13T07:02:43.632145 WARN POST /login"],
  "input_tag":"syslog.host1",
  "input_tag_last":"host1",
}

Output message by joining with a delimiter

As default, the grepcounter plugin outputs matched message as an array as shown above. You may want to output message as a string, then use delimiter option like:

<match syslog.**>
  type grepcounter
  count_interval 60
  input_key message
  regexp WARN
  exclude favicon.ico
  threshold 1
  add_tag_prefix warn.count
  delimiter \n
</match>

Then, output bocomes as belows (indented). You can see the message field is joined with \n.

warn.count.syslog.host1: {
  "count":2,
  "message":"20.4.01/13T07:02:13.232645 WARN POST /auth\n20.4.01/13T07:02:43.632145 WARN POST /login",
  "input_tag":"syslog.host1",
  "input_tag_last":"host1",
}

Parameters

count_interval

The interval time to count in seconds. Default is 60.
input_key field_key

The target field key to grep out. Use with regexp or exclude.
regexp regexp

The filtering regular expression
exclude regexp

The excluding regular expression like grep -v
regexp[1-20] field_key regexp (experimental)

The target field key and the filtering regular expression to grep out. No message is outputted in this case.
exclude[1-20] field_key regexp (experimental)

The target field key and the excluding regular expression like grep -v. No message is outputted in this case.
threshold

The threshold number to emit. Emit if count value >= specified value.
greater_equal

This is same with threshold option. Emit if count value is greater than or equal to (>=) specified value.
greater_than

Emit if count value is greater than (>) specified value.
less_than

Emit if count value is less than (<) specified value.
less_equal

Emit if count value is less than or equal to (<=) specified value.
tag

The output tag. Required for aggregate all.
add_tag_prefix

Add tag prefix for output message
remove_tag_prefix

Remove tag prefix for output message
add_tag_suffix

Add tag suffix for output message
remove_tag_suffix

Remove tag suffix for output message

remove_tag_slice min..max

Remove tag parts by slice function. FYI: This option behaves like tag.split('.').slice(min..max).

For example,
```
  remove_tag_slice 0..-2
```
changes an input tag foo.bar.host1 to foo.bar.
aggregate

Aggregation unit. One of all, in_tag, out_tag can be specified. Default is all.
- all counts summation for all input messages and emit one message in each interval.
- in_tag counts summation for each input tag seperately.
- out_tag counts summation for each tag modified by add_tag_prefix, remove_tag_prefix, or remove_tag_slice.

delimiter

Output matched messages after joined with the specified delimiter.
replace_invalid_sequence

Replace invalid byte sequence in UTF-8 with '?' character if true
store_file

Store internal count data into a file of the given path on shutdown, and load on statring.

ChangeLog

See CHANGELOG.md for details.

Contributing

Fork it
Create your feature branch (git checkout -b my-new-feature)
Commit your changes (git commit -am 'Add some feature')
Push to the branch (git push origin my-new-feature)
Create new Pull Request