Fluent ECS is a filter plugin for fluentbit that aims to transform logs of various sources to the field schema defined by the Elastic Common Schema.
works on my machine
Log transformations for application are added as required for a specific logging setup. The transformation rules are not exhaustive. They do not cover every log that may be produced by the supported applications.
Because of the fluent-bit issue #8156 Rust filters like fluent-ecs do not seem to work with the size of real world logs at the moment. When manually fixing and compiling fluent-bit as described there in the comments, fluent-ecs will work. Issue #7112 points out that WASM filters result in high CPU usage. That means that the WASM support in fluent-bit is pretty useless in its current state.
Add the target for web assembly.
rustup target add wasm32-unknown-unknown
Build with cargo.
cargo build --target wasm32-unknown-unknown --release
Fluent ECS tries to provide normalized values for the field field.severity across different applications.
The following values are used.
| level | severity | +-------+----------| | trace | 50 | | debug | 100 | | info | 200 | | warn | 300 | | error | 400 |
Information added by the Kubernetes Plugin are converted to the ECS scheme.
Fluent ECS tries to detect the application that produced logs in order do convert these logs app-specifically. At the moment the application detection is based on evaluating labels and annotations added by the fluent-bit Kubernetes plugin. The following annotations and labels are evaluated. The first that matches a keyword for a supported applications determines how the log event is processed further.
- Annotation: fluent-ecs.bieniek-it.de/parser
- Label: app.kubernetes.io/name
- Label: component
- Keyword: etcd
Etcd logs in JSON format. The fluent-ecs support for etcd moves JSON fields unknown in ECS to a single array "misc". This way the log index is not cluttered with to much too etcd-specific fields.
- Keyword: etcd
Metallb logs in JSON format. The fluent-ecs support for Metallb moves JSON fields unknown in ECS to a single array "misc". This way the log index is not cluttered with to much too Metallb-specific fields.
- Keyword: kubernetes-dashboard-metrics-scraper
Some of the logs of the metrics scraper are JSON logs. The keys are converted to the correct ECS keys.
- Keyword: postfix
Postfix Logs in plain text. fluent-ecs will parse these plain text logs and will extract information about network connections and transferred mails.
- Keyword: keycloak
Expects the logs to be JSON format.
This can be achieved e.g. by setting the environment varible KC_LOG_CONSOLE_OUTPUT to json.
Login events are parsed too.
They are logged with the logger org.keycloak.events on level debug.
Setting only this logger to level debug while keeping the rest on level info can be achieved e.g. by setting the environment variable KC_LOG_LEVEL to info,org.keycloak.events:debug.