Structured Logs Search (slearch)
This is a simple utility to search through structured JSON or text logs that come out of your logging systems; eg: kubectl logs
or docker logs
.
Slearch will read structured JSON or text logs via stdin
that are separated by newlines and allow you to filter out exact or regex matches per log line. It will print the matching results in the same order they are read.
By default it will match queries as AND
, meaning it will only show results where all queries for a line match. This can be changed to an OR
query with the -s
flag.
Matched queries can either be exact with -m
or by regex (using golang stdlib regexp package) with -r
.
It will attempt to autodetect what format the log line is in, and match based on that format. If you want to specify a particular format you can with the -t
argument.
Installing
$ go get -u github.com/vishen/go-slearch
Installing from source
$ make build
Usage
Read stuctured logs from STDIN and filter out lines based on exact and regex matches. Currently only supports JSON and text logs.
Usage:
go-slearch [flags]
Flags:
-h, --help help for go-slearch
-d, --key_delimiter string the string to split the key on for complex key queries
-m, --match strings key and value to match on. eg: label1=value1
-p, --print_keys strings keys to print if a match is found
-r, --regexp strings key and value to regex match on. eg: label1=value*
-s, --search_type string the search type to use: 'and' or 'or' (default "and")
-t, --type string the log type to use: 'json' or 'text'. If unspecified it will attempt to use all log types
-v, --verbose verbose output
Example
$ cat example.txt
{"severity": "info", "key1": "value1", "key2": "value2"}
{"severity": "debug", "key1": "value12", "key2": "value2", "key3": "value3"}
{"severity": "info", "key1": "value13", "key3": "value3"}
{"severity": "info", "key1": "value14", "key4": "value4"}
{"severity": "info", "key2": "value2", "key5": 5}
{"key2": "value2", "complexKey": {"key1": "value1", "complexKey1": {"key3": "value3"}}}
# Exact match search with -m
$ cat example.txt | go-slearch -m key1="value1"
{"severity": "info", "key1": "value1", "key2": "value2"}
# Regex match search with -r
$ cat example.txt | go-slearch -r key1="value1"
{"severity": "debug", "key1": "value12", "key2": "value2", "key3": "value3"}
{"severity": "info", "key1": "value13", "key3": "value3"}
{"severity": "info", "key1": "value14", "key4": "value4"}
{"severity": "info", "key1": "value1", "key2": "value2"}
# Exact match on multiple keys, but because by default it queries with an AND we get no results
$ cat example.txt | go-slearch -m key1=value1,key3=value3
# Exact match on multiple key WITH OR
$ cat example.txt | go-slearch -m key1=value1,key3=value3 -s or
{"severity": "debug", "key1": "value12", "key2": "value2", "key3": "value3"}
{"severity": "info", "key1": "value1", "key2": "value2"}
{"severity": "info", "key1": "value13", "key3": "value3"}
# Search on a complex key, the -d is the delimiter to use for a complex key, which is '.' in this case
$ cat example.txt | go-slearch -m complexKey.key1=value1 -d .
{"key2": "value2", "complexKey": {"key1": "value1", "complexKey1": {"key3": "value3"}}}
# Only print certain keys
$ cat example.txt | go-slearch -m key1=value1,key3=value3 -s or -p severity,key1,key2
{"severity":"debug", "key1":"value12", "key2":"value2"}
{"severity":"info", "key1":"value1", "key2":"value2"}
{"severity":"info", "key1":"value13"}
TODO
- Bold what was matched
- Sort based on keys
- Tests and documentation and examples
- Document regex case insensitive match ("(?i)" a the start of the regex)
- Add new formatters:
- Add log formatter that splits stuff by delimeter, and you can the select which element(s) via $n
- Add googles glog format
- Better controlled concurrency
- Ignore parts of the line that don't match a specified format?
- Add integration support (might not be worth it if we have to pull in the client libraries for these)?
- kubernetes
- docker
- stackdriver logs