This script parses logs in the Apache Common Log Format (CLF) used by Apache and NGINX and stores them without personal information as JSON, CSV or in a database table.
The tool can be installed via pip. Usually you want to create a virtual environment first, but this is optional.
# optional virtual env
python3 -m venv env
source env/bin/activate
pip install git+https://github.com/jochenklar/logs
In order to resolve countries from IP addresses, goto https://dev.maxmind.com/geoip/geoip2/geolite2/, create an account, and download the GeoLite2-Country.mmdb
database.
The tool has several options which can be inspected using the help option -h, --help
:
usage: logparser [-h] [--format {json,csv,sql}] [--host HOST]
[--database DATABASE] [--geoip2-database GEOIP2_DATABASE]
[--ignore-host IGNORE_HOST] [--ignore-path IGNORE_PATH]
[--log-level LOG_LEVEL] [--log-file LOG_FILE]
[--config-file CONFIG_FILE]
path
positional arguments:
path Path to the file to process, can be a pattern using *
optional arguments:
-h, --help show this help message and exit
--format {json,csv,sql}
Output format, default: json
--host HOST Host for this log, useful if logs of multiple hosts
are aggregated in one place, default: localhost
--database DATABASE Database connection string, e.g. postgresql+psycopg2:/
/username:password@host:port/dbname
--geoip2-database GEOIP2_DATABASE
Path to the geoip2 database
--ignore-host IGNORE_HOST
Host in the logs to be ignored, useful for internal
ips, can be repeated
--ignore-path IGNORE_PATH
Path in the logs to be ignored, useful for recurring
API calls, can be repeated
--log-level LOG_LEVEL
Log level (ERROR, WARN, INFO, or DEBUG)
--log-file LOG_FILE Path to the log file
--config-file CONFIG_FILE
File path of the config file
The only mandatory argument is the PATH
to the logfile to process. The optional arguments can be provided on the command line, but also:
- (in upper case) as environment variables, e.g.
FORMAT=csv
- from
.env
file in the directory from where the script is called (with the same syntax) - in a config file given by
--config-file
, or located atlogparser.conf
,~/.logparser.conf
, or/etc/logparser.conf
.
In order to connect to a database connection string DATABASE
has to be provided and psycopg2-binary
or mysqlclient
have to be installed.
Examples:
logparser /var/log/apache2/access.log
logparser /var/log/apache2/access.log --format=json
logparser /var/log/apache2/access.log --format=sql --log-level=info --host=example.com
The script is intended to be used with logrotate. With the following setup logs are parsed and ingested into the database before they are rotated. Lets assume a Apache2 setup on Ubuntu/Debian with two virtual hosts, producing example1.com.access.log
and example2.com.access.log
, and the following /etc/logrotate.d/apache2
:
/var/log/apache2/*.log {
...
prerotate
/etc/logparser-prerotate.sh
endscript
}
This configuration calls the script /etc/logparser-prerotate.sh
before rotating the logs. /etc/logparser-prerotate.sh
should have the following content:
#!/bin/bash
/path/to/logparser/env/bin/logparser /var/log/example1.com.access.log --host=example1.com
/path/to/logparser/env/bin/logparser /var/log/example2.com.access.log --host=example2.com
and logparser can be configured like this in /etc/logparser.conf
:
[default]
database = postgresql+psycopg2://username:password@host:port/dbname
format = sql
geoip2_database = /path/to/GeoLite2-Country.mmdb