High performant transaction logger for go
This library is compatible with Go 1.12+
Please refer to CHANGELOG.md
if you encounter breaking changes.
Introduction
The goal of this project is to provide hyper performant, zero memory allocation transaction logger, that can work with local and cloud file storage. Tapper logger allow log rotation, where on each rotation additional logic can be delegated to the external web service or a shell script.
Usage
cfg := &config.Stream{
URL: "/tmp/logfile.log",
Rotation: &config.Rotation{
EveryMs: 20000,
URL: "s3://my.bucket/data/logfile.log.[yyyyMMdd_HH]-%v",
},
}
logger, err := log.New(cfg, "myID", afs.New())
if err != nil {
slog.Fatal(err)
}
provider := msg.NewProvider(2048, 32)
for i :=0;i<100;i++ {
message := provider.NewMessage()
message.PutString("k1", "value1")
message.PutInt("k2", 2)
message.PutStrings("k3", []string{"1", "3"})
err = logger.Log(message)
if err != nil {
slog.Fatal(err)
}
message.Free()
}
logger.Close()
Configuration
-
URL: location of main log stream
-
FlushMod: optional flush frequency (testing only, do not use on production)
-
Codec: optional compression codec (gzip) of main stream not recommended for production, compressing on rotation is much faster)
-
Rotation: optional rotation config where:
- EveryMs: rotation frequency in ms
- Codec: optional compression codec (gzip) applied on log rotation.
- URL: rotation dest pattern
- Emit: optional rotation event notification vi URL or OS process (shell command)
- URL URL to call with specified parameters
- Params URL parameters (query string)
- Name: name of command to run
- Args: command arguments
- in Args or Params values you can use the following variables:
- $DestPath variable to refer to rotated absolute file name
- $Dest to expand with dest URL
- $DestName to expand with simple file name
- $TimePath yyyy/mm/dd/hh rotation create time base path fragment
Rotation URL pattern expression
- time expression placed in squere brackets: [yyyy-MM-dd_HH]
- logger ID - rotation seqence: %v
Configuring rotation event with 3rd party web service
The following configuration drives rotation notification on http://127.0.0.1:8083 vi REST service as GET request.
URL: /opt/app/logs/datastream1.log
FlushMod: 1
Rotation:
EveryMs: 30000
URL: /opt/app/logs/datastream1.log.[yyyy-MM-dd_hh-mm-ss].%v
Emit:
URL: http://127.0.0.1:8083/log/datastream1
Params:
DestPath: $DestPath
DestName: $DestName
TimePath: $TimePath
See event consumer service example.
Configuring rotation event with a shell script
URL: /opt/app/logs/datastream1.log
FlushMod:
Rotation:
EveryMs: 10000
URL: /opt/app/logs/datastream1.log.[yyyy-MM-dd_hh-mm-ss].%v
Emit:
Command: /bin/bash
Args:
- /opt/mediator/script/sitelet.sh
- $DestPath
- $DestName
- $TimePath
Messages
To reduce log message memory overhead, a message can be created by Provider, which handles data pooling. You can create a log message with the following snippet using json format.
provider := msg.NewProvider(avgMessageSize, concurrency, json.New)
message := provider.NewMessage()
defer message.Free()
One message is no longer needed Free method returns it back to the provider pool.
Log message support primitive and complex data structure.
meesage.Put([]byte{`"k1":"raw data"`})
meesage.PutString("k2", "v2")
meesage.PutNonEmptyString("k2.1", v)
meesage.PutB64EncodedBytes("k2.2", rawData)
meesage.PutInt("k3", 3)
meesage.PutFloat("k3", 3.2)
meesage.PutBool("k3", false)
meesage.PutInts("k4", []int{1,2,3})
meesage.PutObject("k5", object)
meesage.PutObjects("k6", objects)
Message Provider also support CSV type . In this case, the message provider constructor takes CSV message type.
provider := msg.NewProvider(avgMessageSize, concurrency, csv.New)
message := provider.NewMessage()
defer message.Free()
Only primitive and slice data types are supported in CSV message.
Benchmark
Benchmark builds b.T x 1K message with 10 attrs and writes the log stream.
BenchmarkLogger_Log
BenchmarkLogger_Log-16 258068 4348 ns/op 0 B/op 0 allocs/op
BenchmarkLogger_Log_Rotation
BenchmarkLogger_Log_Rotation-16 211363 5318 ns/op 5 B/op 0 allocs/op
License
The source code is made available under the terms of the Apache License, Version 2, as stated in the file LICENSE
.
Individual files may be made available under their own specific license, all compatible with Apache License, Version 2. Please see individual files for details.
Credits and Acknowledgements
Library Author: Adrian Witas