Logster is a utility for reading log files and generating metrics in Graphite or Ganglia. It is ideal for visualizing trends of events that are occurring in your application/system/error logs. For example, you might use logster to graph the number of occurrences of HTTP response code that appears in your web server logs. Logster maintains a cursor, via logtail, on each log file that it reads so that each successive execution only inspects new log entries. In other words, a 1 minute crontab entry for logster would allow you to generate near real-time trends in Graphite or Ganglia for anything you want to measure from your logs. This tool is made up of a framework script, logster, and parsing scripts that are written to accommodate your specific log format. Two sample parsers are included in this distribution. The parser scripts essentially read a log file line by line, apply a regular expression to extract useful data from the lines you are interested in, and then aggregate that data into metrics that will be submitted to either Ganglia or Graphite. Take a look through the sample parsers, which should give you some idea of how to get started writing your own. History The logster project was created at Etsy as a fork of ganglia-logtailer (https://bitbucket.org/maplebed/ganglia-logtailer). We made the decision to fork ganglia-logtailer because we were removing daemon-mode from the original framework. We only make use of cron-mode, and supporting both cron- and daemon-modes makes for more work when creating parsing scripts. We care strongly about simplicity in writing parsing scripts -- which enables more of our engineers to write log parsers quickly. Installation Logster depends on the "logtail" utility that can be obtained from the logcheck package, either from a Debian package manager or from source: http://packages.debian.org/source/sid/logcheck. An RPM for logtail can be found here: http://rpmfind.net/linux/RPM/epel/testing/5/x86_64/logcheck-1.3.13-3.el5.noarch.html Once you have logtail installed, then the only other thing you need to do is run the installation commands in the Makefile: $ sudo make install Usage You can test logster from the command line. There are two sample parsers (SampleGangliaLogster and SampleGraphiteLogster) that can be used to generate stats from an Apache access log. The --dry-run option will allow you to see the metrics being generated on stdout rather than sending them to either Ganglia or Graphite. $ sudo /usr/sbin/logster --dry-run SampleGangliaLogster /var/log/httpd/access_log $ sudo /usr/sbin/logster --dry-run SampleGraphiteLogster /var/log/httpd/access_log Additional usage details can be found with the -h option: $ ./logster -h usage: logster [options] parser logfile Tail a log file and filter each line to generate metrics that can be sent to common monitoring packages. options: -h, --help show this help message and exit -p METRIC_PREFIX, --metric-prefix=METRIC_PREFIX Add prefix to all published metrics. This is for people that may multiple instances of same service on same host. --gmetric-options=GMETRIC_OPTIONS Options to pass to gmetric such as -d 180 -c /etc/ganglia/gmond.conf (default). These are passed directly to gmetric. --graphite-host=GRAPHITE_HOST Hostname and port for Graphite collector, e.g. graphite.example.com:2003 -s STATE_DIR, --state-dir=STATE_DIR Where to store the logtail state file. Default location /var/run -d, --dry-run Parse the log file but send stats to standard output. -D, --debug Provide more verbose logging for debugging.