s4
)
Super Speedy Syslog Searcher! (Speedily search and merge log messages by datetime.
Super Speedy Syslog Searcher (s4
) is a command-line tool to search
and merge varying log messages from varying log files, sorted by datetime.
This includes log files that are compressed (.gz
, .xz
), or archived (.tar
),
and binary format logs including utmpx user accounting records (utmp
, wtmp
),
systemd journal logs (.journal
), and Microsoft Event Logs (.evtx
).
It will parse a variety of formal and ad-hoc log message datetime formats.
Datetime filters may be passed to narrow the search to a datetime range.
The first goal of s4
is speedy searching and printing.
Use
super_speedy_syslog_searcher
Install Assuming rust is installed, run
cargo install super_speedy_syslog_searcher
s4
Run For example, print all the log messages in syslog files under /var/log/
s4 /var/log
On Windows, print the ad-hoc logs under C:\Windows\Logs
s4.exe C:\Windows\Logs
Or the Windows Event logs
s4.exe C:\Windows\System32\winevt\Logs
Print the log messages after January 1, 2022 at 00:00:00
s4 /var/log -a 20220101
Print the log messages from January 1, 2022 00:00:00 to January 2, 2022
s4 /var/log -a 20220101 -b 20220102
or
s4 /var/log -a 20220101 -b @+1d
Print the log messages on January 1, 2022, from 12:00:00 to 16:00:00
s4 /var/log -a 20220101T120000 -b 20220101T160000
Print only the log messages since yesterday at this time
s4 /var/log -a=-1d
Print only the log messages that occurred two days ago
(with the help of GNU date
)
s4 /var/log -a $(date -d "2 days ago" '+%Y%m%d') -b @+1d
Print only the log messages that occurred two days ago during the noon hour
(with the help of GNU date
)
s4 /var/log -a $(date -d "2 days ago 12" '+%Y%m%dT%H%M%S') -b @+1h
Print only the log messages that occurred two days ago during the noon hour in
Bengaluru, India (timezone offset +05:30) and prepended with equivalent UTC
datetime (with the help of GNU date
)
s4 /var/log -u -a $(date -d "2 days ago 12" '+%Y%m%dT%H%M%S+05:30') -b @+1h
--help
Speedily search and merge log messages by datetime.
DateTime filters may be passed to narrow the search.
It aims to be very fast.
Usage: s4 [OPTIONS] <PATHS>...
Arguments:
<PATHS>... Path(s) of log files or directories.
Directories will be recursed. Symlinks will be followed.
Paths may also be passed via STDIN, one per line. The user must
supply argument "-" to signify PATHS are available from STDIN.
Options:
-a, --dt-after <DT_AFTER>
DateTime Filter After: print syslog lines with a datetime that is at
or after this datetime.
For example, "20200102T120000" or "-5d".
-b, --dt-before <DT_BEFORE>
DateTime Filter Before: print syslog lines with a datetime that is at
or before this datetime.
For example, "2020-01-03T23:00:00.321-05:30" or "@+1d+11h"
-t, --tz-offset <TZ_OFFSET>
Default timezone offset for datetimes without a timezone.
For example, log message "[20200102T120000] Starting service" has a
datetime substring "20200102T120000".
The datetime substring does not have a timezone offset
so the TZ_OFFSET value would be used.
Example values, "+12", "-0800", "+02:00", or "EDT".
To pass a value with leading "-" use "=" notation, e.g. "-t=-0800".
If not passed then the local system timezone offset is used.
[default: -07:00]
-z, --prepend-tz <PREPEND_TZ>
Prepend a DateTime in the timezone PREPEND_TZ for every line.
Used in PREPEND_DT_FORMAT.
-u, --prepend-utc
Prepend a DateTime in the UTC timezone offset for every line.
This is the same as "--prepend-tz Z".
Used in PREPEND_DT_FORMAT.
-l, --prepend-local
Prepend DateTime in the local system timezone offset for every line.
This is the same as "--prepend-tz +XX" where +XX is the local system
timezone offset.
Used in PREPEND_DT_FORMAT.
-d, --prepend-dt-format <PREPEND_DT_FORMAT>
Prepend a DateTime using the strftime format string.
If PREPEND_TZ is set then that value is used for any timezone offsets,
i.e. strftime "%z" "%:z" "%Z" values, otherwise the timezone offset value
is the local system timezone offset.
[Default: %Y%m%dT%H%M%S%.3f%z]
-n, --prepend-filename
Prepend file basename to every line.
-p, --prepend-filepath
Prepend file full path to every line.
-w, --prepend-file-align
Align column widths of prepended data.
--prepend-separator <PREPEND_SEPARATOR>
Separator string for prepended data.
[default: :]
--separator <LOG_MESSAGE_SEPARATOR>
An extra separator string between printed log messages.
Per log message not per line of text.
Accepts a basic set of backslash escape sequences,
e.g. "\0" for the null character.
--journal-output <JOURNAL_OUTPUT>
The format for .journal file log messages.
Matches journalctl --output options.
[default: short]
[possible values: short, short-precise, short-iso, short-iso-precise,
short-full, short-monotonic, short-unix, verbose,
export, cat]
-c, --color <COLOR_CHOICE>
Choose to print to terminal using colors.
[default: auto]
[possible values: always, auto, never]
--blocksz <BLOCKSZ>
Read blocks of this size in bytes.
May pass value as any radix (hexadecimal, decimal, octal, binary).
Using the default value is recommended.
Most useful for developers.
[default: 65535]
-s, --summary
Print a summary of files processed to stderr.
Most useful for developers.
-h, --help
Print help
-V, --version
Print version
DateTime Filters may be strftime specifier patterns:
"%Y%m%dT%H%M%S*"
"%Y-%m-%d %H:%M:%S*"
"%Y-%m-%dT%H:%M:%S*"
"%Y/%m/%d %H:%M:%S*"
"%Y%m%d"
"%Y-%m-%d"
"%Y/%m/%d"
"+%s"
Each * is an optional trailing 3-digit fractional sub-seconds,
or 6-digit fractional sub-seconds, and/or timezone.
Pattern "+%s" is Unix epoch timestamp in seconds with a preceding "+".
For example, value "+946684800" is be January 1, 2000 at 00:00, GMT.
DateTime Filters may be custom relative offset patterns:
"+DwDdDhDmDs" or "-DwDdDhDmDs"
"@+DwDdDhDmDs" or "@-DwDdDhDmDs"
Custom relative offset pattern "+DwDdDhDmDs" and "-DwDdDhDmDs" is the offset
from now (program start time) where "D" is a decimal number.
Each lowercase identifier is an offset duration:
"w" is weeks, "d" is days, "h" is hours, "m" is minutes, "s" is seconds.
For example, value "-1w22h" is one week and twenty-two hours in the past.
Value "+30s" is thirty seconds in the future.
Custom relative offset pattern "@+DwDdDhDmDs" and "@-DwDdDhDmDs" is relative
offset from the other datetime.
Arguments "-a 20220102 -b @+1d" are equivalent to "-a 20220102 -b 20220103".
Arguments "-a @-6h -b 20220101T120000" are equivalent to
"-a 20220101T060000 -b 20220101T120000".
Without a timezone, the Datetime Filter is presumed to be the local system
timezone.
Command-line passed timezones may be numeric timezone offsets,
e.g. "+09:00", "+0900", or "+09", or named timezone offsets, e.g. "JST".
Ambiguous named timezones will be rejected, e.g. "SST".
--prepend-tz and --dt-offset function independently:
--dt-offset is used to interpret processed log message datetime stamps that
do not have a timezone offset.
--prepend-tz affects what is pre-printed before each printed log message line.
--separator accepts backslash escape sequences:
"\0","\a","\b","\e","\f","\n","\r","\\","\t","\v",
Resolved values of "--dt-after" and "--dt-before" can be reviewed in
the "--summary" output.
DateTime strftime specifiers are described at
https://docs.rs/chrono/latest/chrono/format/strftime/
DateTimes supported are only of the Gregorian calendar.
DateTimes supported language is English.
Is s4 failing to parse a log file? Report an Issue at
https://github.com/jtmoon79/super-speedy-syslog-searcher/issues/new/choose
About
s4
?
Why Super Speedy Syslog Searcher (s4
) is meant to aid Engineers in reviewing
varying log files in a datetime-sorted manner.
The primary use-case is to aid investigating problems wherein the time of
problem occurrence is known but otherwise there is little source evidence.
Currently, log file formats vary widely. Most logs are an ad-hoc format. Even separate log files on the same system for the same service may have different message formats! 😵 Sorting these logged messages by datetime may be prohibitively difficult. The result is an engineer may have to "hunt and peck" among many log files, looking for problem clues around some datetime; so tedious!
Enter Super Speedy Syslog Searcher 🦸 ‼
s4
will print log messages from multiple log files in datetime-sorted order.
A "window" of datetimes may be passed, to constrain the period of printed
messages. This will assist an engineer that, for example, needs to view all
log messages that occurred two days ago between 12:00 and 12:05 among log files taken from multiple
systems.
The ulterior motive for Super Speedy Syslog Searcher was the primary developer wanted an excuse to learn rust 🦀, and wanted to create an open-source tool for a recurring need of some Software Test Engineers 😄
See the real-world example rationale in section
logging chaos; the problem s4
solves.
Features
- Parses:
- Ad-hoc log messages using formal datetime formats:
- Red Hat Audit Log files
- binary "utmpx" user accounting records files
- binary Windows Event Log files
- binary systemd journal files with printing options matching
journalctl
- many varying text log messages with ad-hoc datetime formats
- multi-line log messages
- Inspects
.tar
archive files for parseable log files *** - Inspects
.gz
and.xz
compressed files for parseable log files *** - Tested against "in the wild" log files from varying sources
(see project path
./logs/
) - Prepends datetime and file paths, for easy programmatic parsing or visual traversal of varying log messages
- Comparable speed as GNU
grep
andsort
(see project tool./tools/compare-grep-sort.sh
; run in github Actions, Job runs4
, Step Run script compare-grep-sort) - Processes invalid UTF-8
- Accepts arbitrarily large files ****
Limitations
- Only processes UTF-8 or ASCII encoded syslog files. (Issue #16)
- Cannot process multi-file
.gz
files (only processes first stream found). (Issue #8) - Cannot process multi-file
.xz
files (only processes first stream found). (Issue #11) - *** Cannot process archive files or compressed files within other
archive files or compressed files (Issue #14),
e.g.logs.tgz
e.g. filesyslog.xz
file within archivelogs.tar
- Cannot process
.zip
archives (Issue #39) - ** ISO 8601
- ISO 8601 forms recognized
(using ISO descriptive format)
YYYY-MM-DDThh:mm:ss
YYYY-MM-DDThhmmss
YYYYMMDDThhmmss
(may use date-time separator character'T'
or character blank space' '
)
- ISO 8601 forms not recognized:
- Absent seconds
- Ordinal dates, i.e. "day of the year", format
YYYY-DDD
, e.g."2022-321"
- Week dates, i.e. "week-numbering year", format
YYYY-Www-D
, e.g."2022-W25-1"
- times without minutes and seconds (i.e. only
hh
)
- ISO 8601 forms recognized
(using ISO descriptive format)
- **** Only for unarchived, uncompressed files (Issue #9, Issue #12, Issue #13) and not for EVTX files (Issue #86)
Hacks
- Entire
.xz
files are read into memory during the initialopen
(Issue #12) - Entire
.evtx
files are read into memory before printing (Issue #86)
More
Building locally
Building on Linux requires:
rust
minimal or moregcc
(which should installcc
,libc
, andlibc-headers
)
From the git cloned project directory run cargo build
.
.journal
files
Parsing Requires libsystemd
to be installed to then use libsystemd.so
.
Requesting Support For DateTime Formats; your particular log file
If you have found a log file that Super Speedy Syslog Searcher does not parse then you may create a new Issue type Feature request (datetime format).
Here is an example user-submitted Issue.
"syslog" and other project definitions
syslog
In this project, the term "syslog" is used generously to refer to any log message that has a datetime stamp on the first line of log text.
Technically, "syslog" is defined among several RFCs proscribing fields, formats, lengths, and other technical constraints. In this project, the term "syslog" is interchanged with "log".
The term "sysline" refers to a one log message which may comprise multiple text lines.
See docs section Definitions of data for more project definitions.
log message
A "log message" is a single log entry for any type of logging scheme; an entry in a utmpx file, an entry in a systemd journal, an entry in a Windows Event Log, a formal syslog message, or an ad-hoc log message.
s4
solves
logging chaos; the problem In practice, most log file formats are an ad-hoc format that may not follow any formal definition. Sorting varying log messages by datetime is prohibitively tedious.
The following real-world example log files are available in project directory
./logs
.
open-source software examples
nginx webserver
For example, the open-source nginx web server
logs access attempts in an ad-hoc format in the file access.log
192.168.0.115 - - [08/Oct/2022:22:26:35 +0000] "GET /DOES-NOT-EXIST HTTP/1.1" 404 0 "-" "curl/7.76.1" "-"
which is an entirely dissimilar log format to the neighboring nginx log file,
error.log
2022/10/08 22:26:35 [error] 6068#6068: *3 open() "/usr/share/nginx/html/DOES-NOT-EXIST" failed (2: No such file or directory), client: 192.168.0.115, server: _, request: "GET /DOES-NOT-EXIST HTTP/1.0", host: "192.168.0.100"
nginx is following the bad example set by the apache web server.
Debian 11
Here is a log snippet from a Debian 11 host, file /var/log/alternatives.log
:
update-alternatives 2022-10-10 23:59:47: run with --quiet --remove rcp /usr/bin/ssh
And a snippet from the same Debian 11 host, file /var/log/dpkg.log
:
2022-10-10 15:15:02 upgrade gpgv:amd64 2.2.27-2 2.2.27-2+deb11u1
And a snippet from the same Debian 11 host, file /var/log/kern.log
:
Oct 10 23:07:16 debian11-b kernel: [ 0.10034] Linux version 5.10.0-11-amd64
And a snippet from the same Debian 11 host,
file /var/log/unattended-upgrades/unattended-upgrades-shutdown.log
:
2022-10-10 23:07:16,775 WARNING - Unable to monitor PrepareForShutdown() signal, polling instead.
commercial software examples
Commercial software and computer hardware vendors nearly always use ad-hoc log message formatting that is even more unpredictable among each log file on the same system.
Synology DiskStation
Here is a log snippet from a Synology DiskStation package DownloadStation:
2019/06/23 21:13:34 (system) trigger DownloadStation 3.8.13-3519 Begin start-stop-status start
And a snippet from a Synology DiskStation OS log file sfdisk.log
on the same
host:
2019-04-06T01:07:40-07:00 dsnet sfdisk: Device /dev/sdq change partition.
And a snippet from a Synology DiskStation OS log file synobackup.log
on the
same host:
info 2018/02/24 02:30:04 SYSTEM: [Local][Backup Task Backup1] Backup task started.
(yes, those are tab characters)
Microsoft Windows 10
Here are is a snippet from a Windows 10 host, log file
${env:SystemRoot}\debug\mrt.log
Microsoft Windows Malicious Software Removal Tool v5.83, (build 5.83.13532.1)
Started On Thu Sep 10 10:08:35 2020
And a snippet from the same Windows 10 host, log file
${env:SystemRoot}\comsetup.log
COM+[12:24:34]: ********************************************************************************
COM+[12:24:34]: Setup started - [DATE:05,27,2020 TIME: 12:24 pm]
COM+[12:24:34]: ********************************************************************************
And a snippet from the same Windows 10 host, log file
${env:SystemRoot}\DirectX.log
11/01/19 20:03:40: infinst: Installed file C:\WINDOWS\system32\xactengine2_1.dll
And a snippet from the same Windows 10 host, log file
${env:SystemRoot}/Microsoft.NET/Framework/v4.0.30319/ngen.log
09/15/2022 14:13:22.951 [515]: 1>Warning: System.IO.FileNotFoundException: Could not load file or assembly
And a snippet from the same Windows 10 host, log file
${env:SystemRoot}/Performance/WinSAT/winsat.log
68902359 (21103) - exe\logging.cpp:0841: --- START 2022\5\17 14:26:09 PM ---
68902359 (21103) - exe\main.cpp:4363: WinSAT registry node is created or present
(yes, it reads hour 14
, and PM
… 🙄)
Summary
This chaotic logging approach is typical of commercial and open-source software. And it's a mess! Attempting to sort log messages by their natural sort mechanism, a datetime stamp, is difficult to impossible.
Hence the need for Super Speedy Syslog Searcher! 🦸