mbivert/log-tools

Introduction

Small tools to help parse OpenBSD's httpd(8) logs (forwarded format).

The code is mostly split in:

1. logtweak: parses one or multiple log files (access.log), eventually filtering them on date, localizing IPs, etc. It outputs a table-like format, with a special character as column-separator;
2. logtag: reads logtweak's output, and, according to a set of rules described in a JSON file, will tag each log line.

Additionally, iploc geolocalizes IPs from the CLI.

Finally, ad hoc scripts can be written to generate basic analytics, identify/ban nefarious IPs, etc.

IP location

Performed using IP databases from ip2location.com.

While it can't be openly redistributed, you can still download a free "LITE" version from their website. See the get-ip2location.sh script.

Tests

There are two types of tests:

1. Per-go function tests: iplocate_test.go, tweak_test.go;
2. Global tests, in tests/, launched by tests.sh.

All tests are written in a peculiar style, see this blog post for more.

Cross-compilation

The following will create amd64 OpenBSD executables logtag, logtweak and iploc:

(local)$ GOOS=openbsd GOENV=amd64 make all

History

Originally, the code was meant to be a bit more generic, and to be coupled with qsplit(1). However, it turns out that quote injection in user-agent is a thing; a big regexp avoids most of the related issues.

A older prototype in Perl (see old/perl/) demonstrates a more generic approach; it suffers from a major performance hit that I haven't bothered to investigate though.