Logsandra ========= Logsandra is a log management application written in Python and using Cassandra as back-end. Requirements ------------ Cassandra 0.6.x, Python 2.6/2.7 (not tested with Python 3.x) and the following Python modules: * Pylons >= 1.0 * Jinja2 * PyYAML * Pycassa * python-dateutil * CherryPy >= 3.1 * Thrift * ordereddict * Cassandra (not automatically installed during the installation phase) Installation ------------ To install Logsandra you run this command: python setup.py install All Python modules will be automatically installed by that command except for the Cassandra bindings. Configuration ------------- To use Logsandra, you need to configure your Cassandra cluster. Below is the current Keyspace configuration needed by Logsandra: <Keyspaces> <Keyspace Name="logsandra"> <ColumnFamily Name="entries" CompareWith="BytesType" /> <ColumnFamily Name="by_date" CompareWith="LongType" /> <ColumnFamily Name="by_date_data" CompareWith="LongType" /> <ColumnFamily Name="categories" CompareWith="UTF8Type" /> <ReplicaPlacementStrategy>org.apache.cassandra.locator.RackUnawareStrategy</ReplicaPlacementStrategy> <ReplicationFactor>3</ReplicationFactor> <EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointSnitch> </Keyspace> </Keyspaces> (In Cassandra 0.7 and higher you will not need to change the Keyspace configuration manually, but as for now Cassandra 0.7 is in development and not supported by Logsandra) The main configuration file for Logsandra is logsandra.yaml, the format is yaml. Below is a list of configuration variables (including type and optional/required): ident identity of the logsandra server [String, Optional/Discovery] httpd_host host on which to serve the logsandra web service [String, Required] httpd_port port on which to serve the logsandra web service [String, Required] httpd_config name of the pylons configuration file [String] cassandra_host cassandra host to connect too [String, Required] cassandra_port cassandra port to connect too [String, Required] cassandra_timeout in seconds until connection is aborted if no response is delivered [String, Optional] paths list of paths to monitor [List, Optional] path directory or file [String, Required] recursive recursively enter sub directories [True/False, Optional] parser dictionary containing parser name and arguments [Dictionary, Required] name name of the parser [String, Required] ... arguments, see specific parser [Depends, Optional] Logsandra is built on Pylons and therefore a Pylons configuration file should be present for Logsandras web service to work. Usage ----- Logsandra consists of two main scripts, which both acts as a daemon, below is the usage pattern: python logsandra-httpd.py [options] start|stop|restart python logsandra-monitord.py [options] start|stop|restart If you want a list of options invoke either one these commands: python logsandra-httpd.py --help python logsandra-monitord.py --help An example setup is to run logsandra-httpd on one server and logsandra-monitord on all servers producing log files. Logsandra-monitord will send the data to the specified Cassandra Cluster. Create your own parser ---------------------- To create your own Logsandra parser you need to create a python file in this directory: logsandra/monitor/parsers/ The name of the file should be in lower case and end with '.py', an example of a files content (the filename is foo.py): from logsandra.monitor.parsers import BaseParser class FooParser(BaseParser): def parse(self, line, source, data): date = None keywords = [] return self.log_entries.add(date=date, entry=line, source=source, keywords=keywords) A better example is to study how existing parsers work in logsandra/monitor/parsers/ directory.