log_monitor

This is a simple go program to monitor a W3C common formatted log being written to on the local system. It will run until you use ^C to exit.

Known Issues

There is a major limitation in the way that averages are being calculated. Because we are not addressing when there is zero traffic during an interval, we are not calculating a valid average if there is no traffic. Given that this is meant to deal with large amounts of traffic vs we have no traffic, it feels like a useful tradeoff.

Assumptions

There is a webserver running on the local system that is writing W3C common formatted logs that can be read.
There is a place where we can write out a temporary sqlite database. (Defaults to /tmp/log.db)
The end user is familiar with setting up a working go environment to get a working binary

Running

go get to fetch dependencies
go run main.go to start the program
(Optionally) Run ./scripts/gen_traffic.sh to send traffic to the webserver. (Assumes the webserver is listing on localhost:80)
go run main.go --help for more information on options.

Tests

go get -t to test dependencies and test dependencies
go test -v to run the tests

Building

go get to fetch dependencies
go build

Improvements and TODO

Move to logrus to get better output handling.
The parsing does not account for HTTP Status and calculates all requests the same. In the future HTTP status should be broken out, etc.
The parsing is janky at best. Porting https://github.com/xojoc/logparse would probably be best.
There is a bit more testing that could be added, specifically around the parsing.