Log Formatter: Logstash in Golang

Log Formatter is a light-weight and extensible and production-ready framework in golang to process log data like Logstash. It ingests data from input as documents, then each document is processed (filter/drop/enhance) by pipeline, and finally sent to output.

Features

Log Formatter has Agent and Monitor component.

Agent

one to one streaming
dynamic configurable
multilevel native logging and prometheus metric

Monitor

show agent’s current status
modify configuration on the fly

Usage and Example

Standalone Agent

go run agent.go -c config.yml

The config file is in yaml format, which should include input, output, pipeline. Here is an exmaple to forward message from Kafka to Kafka.

input:
    kafka:
        brokers: 
            - "127.0.0.1:9092"
        topic: "test01"
        group_name: "formatter-test"
        version: "2.6.0"
output:
    kafka:
        broker: "127.0.0.1:9092"
        topic: "test02"
pipeline:
    formatters:
        - forwarder:

Agent Plugin

See tutorial on how to write your own plugin. [TODO:]

Native Supported Plugins

Input: console, kafka, es
Output: console, kafka, es
Formatter: parser, filter, forwarder

Monitor: A stateless Web Server with Gin+Vue

Monitor and Agent communicate by grpc.

Deploy agents with monitor

# agent
`go run agent.go -c config.yml -monitor <web-rpc-port> -rpcp <agent-rpc-port>   `
# controller
`go run monitor.go -rpcp <web-rpc-port> -webp <web-port>

You can try to play around with monitor's web interface by running docker-compose -f deployment/docker-compose.yaml up, which will set up a monitor on localhost:8080 with 10 agents.

Docker and K8S deployment

The docker images are available on https://hub.docker.com/u/formatter.