Amulog is a tool to support system log management. The main function is to classify log messages with automatically generated log templates (formats and variable locations), and to store the data in a database. This system works on python3.
- Source: https://github.com/amulog/amulog
- Bug Reports: https://github.com/amulog/amulog/issues
- Author: Satoru Kobayashi
- License: BSD-3-Clause
- Support multiple databases: sqlite and mysql
- Smart log segmentation with log2seq
- Multiple template generation algorithms such as: Drain, SHISO, LenMa, FT-tree, Dlog, etc.
- Support Online (incremental) and Offline (hindsight) use
- Suspend and resume the template generation process
- Import and Export log templates if you need
- Edit log templates manually if you need
- Search API with datetime, hostname and log template IDs
$ pip install amulog
For the first step, save following config as test.conf
on an empty directory.
[general] src_path = logfile.txt src_recur = false logging = auto.log [database] database = sqlite3 sqlite3_filename = log.db [log_template] lt_methods = drain indata_filename = ltgen.dump
Then modify general.src_path
option to a logfile you want to load.
(If you want to use multiple files, change general.src_recur
into true and specify directory name to general.src_path
.)
Try following command to generate database:
$ python -m amulog db-make -c test.conf
$ python -m amulog show-db-info -c test.conf
shows status of the generated database.
$ python -m amulog show-lt -c test.conf
shows all generated log templates in the given logfile.
$ python -m amulog show-log -c test.conf ltid=2
shows all log messages corresponding to log template ID 2.
Try following command to resume generating database:
$ python -m amulog db-add -c test.conf logfile2.txt
Following command exports all log templates in the database:
$ python3 -m amulog show-db-import -c test.conf > exported_tpl.txt
You can modify the exported templates manually.
Note that some special letters (\\
, @
, *
) are escaped in the exported templates.
To import the templates, save following config as test2.conf
.
[general] src_path = logfile.txt src_recur = false logging = new_auto.log [database] database = sqlite3 sqlite3_filename = new_log.db [log_template] lt_methods = import indata_filename = new_ltgen.dump [log_template_import] def_path = exported_tpl.txt
Then, try generating database again:
python -m amulog db-make -c test2.conf
Amulog uses log2seq to parse input log messages in DB generation.
If your data is not a default syslog output format, you need to specify an appropriate log2seq parser script.
The log2seq parser script is specified in manager.parser_script
in amulog config file.
[manager] parser_script = test_parser.py fail_output = fail.log
If the parser fails to parse some of the input log messages,
they are stored in manager.fail_output
file.
You can check this file to test whether the parser is working appropriately or not.
There are example parser scripts in log2seq repository.
see help with following command:
python -m amulog -h
This tool is demonstrated at International Journal of Network Management and CNSM2020.
If you use this code, please consider citing:
@article{Kobayashi_IJNM2022, author = {Kobayashi, Satoru and Yamashiro, Yuya and Otomo, Kazuki and Fukuda, Kensuke}, title = {amulog: A general log analysis framework for comparison and combination of diverse template generation methods*}, journal = {International Journal of Network Management}, volume = {32}, number = {4}, pages = {e2195}, doi = {https://doi.org/10.1002/nem.2195}, year = {2022} } @inproceedings{Kobayashi_CNSM2020, author = {Kobayashi, Satoru and Yamashiro, Yuya and Otomo, Kazuki and Fukuda, Kensuke}, booktitle = {Proceedings of the 16th International Conference on Network and Service Management (CNSM'20)}, title = {amulog: A General Log Analysis Framework for Diverse Template Generation Methods}, pages={1-5}, year = {2020} }