/ade

ADE detects anomalous time slices and messages in Linux logs (either RFC3164 or RFC5424 format) using statistical learning.

Primary LanguageJavaGNU General Public License v3.0GPL-3.0

Anomaly Detection Engine for Linux Logs (ADE)

ADE can process a large numbers of logs from a large number of Linux systems to create a compact summary of those logs. The summary identifies and consolidates similar text strings into a single message example and assigns it a key (message id).  The summary determines if  the message id are being issued when expected, are being issued at the expected rate during a time slice, and how often during the day are the message or a similar message (same message id) issued.

You can use those results to examine

  • A set of logs to find anomalies which may be helpful when attempting to find the root cause of a problem or incident
  • The currently generated logs to find anomalies which may be helpful when attempting to find the cause of an on-going problem or incident

Please see http://openmainframeproject.github.io/ade/ for documentation on ADE.

Releases

Saw Kill 1.0.4

  • Fix problems with timezone in Junit test
  • Fix problems with year of test data in function test

Fall Kill 1.0.3

CII Best Practices

  • Support for Core Infrastructure Initiative
    • add travis-ci build for every pull request
    • analyze every pull request with Sonarqube
    • store results of Sonarqube analysis at Sonarqube.com
  • Add sample to mask sensitive data within Linux logs to allow sharing of logs
  • Fix problem with train_test.sh
  • Fix additional problems identified by Sonarqube

Poesten Kill 1.0.2

  • Support for changing analytics
    • command to check syntax of model (flowlayout.xml file)
    • command to print out statistical information contained within model file (.bin file) to text file
    • command to print out version of code and data base
  • Multiple SonarQube(TM) issues fixed
  • Fix to problem with regression test
  • Wiki article "Example of reading ADE data into R objects"

Esopus Creek 1.0.1

  • Support for MariaDB(TM)
  • Verify script - determine if sufficient messages are available to create a valid model
  • Multiple SonarQube(TM) issues fixed
  • Wiki article "Hints on how to update XSLT - tailor the output shown in a browser to problem"

Initial release 1.0.0

  • Parsing of Linux Logs in RFC5424 and RFC3164 format
  • Splitting logs into time slices
  • Handling wrapper messages
  • Statistical analysis of logs
  • Creates output