python script for convert log files to sqlite based on regexp named groups
- multiline log processing
- documentation :)
> python logressor.py -h
logressor.py 0.4
usage: logressor.py [-h] [--file FILE] [--regexp REGEXP] [--format FORMAT]
[--remove FIELDLIST] [--logtype TYPE] [--list]
[--sqlite SQLITEFILE] [--table [TABLE]] [-v] [-d] [--drop]
[--vacuum]
This script is able to convert log files to sqlite format based
on regexp named group method.
optional arguments:
-h, --help show this help message and exit
--file FILE log file to work on (or standard input, if parameter
not given)
--regexp REGEXP regexp with named groups to separate log values
--format FORMAT format of named groups in parseable dict
--remove FIELDLIST comma separated list of removabel fields
--logtype TYPE predefined log type from logressor.dict or user.dict
--list list predefined log types from logressor.dict or
user.dict
--sqlite SQLITEFILE the result sqlite file name (or standard output, if
parameter not given)
--table [TABLE] the table name in sqlite database
-v, --version
-d, --debug debug (default: False)
--drop drop table before create (default: False)
--vacuum vacuum the database after inserts
Sample usage:
Process sample 1)
python logressor.py \
--file sample/s.log \
--regexp "^(?P<v1>.{15})\s+(?P<v2>\S+)\s+(?P<v3>\S+)*" \
--sqlite sample/output.sqlite \
--format "{'v1':{'type':'timestamp','format':'%b %d %H:%M:%S'},'v2':'real'}" \
--remove "v3" \
--drop
Process sample 2) (copy user.dict-sample to user.dict!)
python logressor.py --file sample/s.log --logtype sample --sqlite sample/output.sqlite --drop
Process appache access.log
cat sample/access.log | python logressor.py --logtype apache --sqlite sample/output.sqlite --drop
List defined log types
python logressor.py --list