/watchlog

monitoring log files and sending parsed logs to a specified URL

Primary LanguagePythonMIT LicenseMIT

watchlog

watchlog is a Python package for monitoring log files and sending parsed logs to a specified URL using HTTP POST requests.

Installation

You can install watchlog using pip:

pip install watchlog

Usage

To use watchlog, you need to create a configuration file in JSON format. Here is an example configuration file:

{
  "check_interval": 5,
  "check": true,
  "files": [
    {
      "path": "/path/to/log/file.log",
      "reg": "^(?P<time>[^ ]*) (?P<level>[^ ]*) (?P<module>[^ ]*) (?P<line>[^ ]*) (?P<message>.*)$",
      "url": "http://example.com/api/logs",
      "latest": true,
      "enable": true,
      "headers": {
        "Content-Type": "application/json"
      },
      "auth": {
        "username": "user",
        "password": "pass"
      }
    }
  ]
}
  • path: The path of the log file to monitor.
  • reg: A regular expression used to parse the log file.
  • url: The URL to send the parsed logs to.
  • latest: Start monitoring from the latest logs.
  • enable: Indicates that this configuration will be enabled, which defaults to true.
  • headers: A dictionary of headers to include in the HTTP POST request.
  • auth: A dictionary with username and password keys for basic authentication.
  • check: if open check ,it will loop check the config file
  • check_interval: circle check interval time(s)

Once you have created the configuration file, you can start monitoring the log file by calling the start() function from the watchlog module:

import asyncio
from watchlog import start

loop = asyncio.get_event_loop()
loop.run_until_complete(start('path/to/config.json'))

You can also use we provide small scripts

watchlog path/to/config/file

#or

watchlog --parse path/to/log_file pattern_str

This will start monitoring the log file specified in the configuration file and sending parsed logs to the specified URL using HTTP POST requests.

example

example use zinc_observe as log search_engine,Illustrates the use of the process

.                     
├── __init__.py       
├── config.json             config file
├── docker-compose.yml      log search engine service
├── log.log                 log file
└── main.py                 main python file to monitor log file

License

watchlog is licensed under the MIT license.