Overview

ll-analyzer or "Large Log files Analyzer" is a simple tool for helping Product Support Engineers to speed up incidents root cause analysis.

This tool origins is back in 2017, it came from the need of dealing with huge log files generated by the SAP Hybris commerce suite. However the ll-analyzer was created with the idea of being extensible to any kind of log structure.

What does it do?

In short:

Load all files (logs) from the directory where is executed.
Analyze concurrently those files looking for "issues"
Summarize in reports all of the issues found.

Why ll-analyzer?

"Yes, we are in the cloud age but we still have to deal with On Prem customer"

Some of them with more than 50 nodes, which means dozens of logs with hundreds of thousands lines.

Then we realize that even with,


$ grep 'europeConsignmentExportJob' *.log > another_log.txt

$ grep 'Exception' *.log > yes_another_log.txt

:75498,75622w! chunk_of_log.txt

\|[0-9]{4,99}\sms\|statement

you will spend some time on those files.

Installation

# mac
env GOOS=darwin GOARCH=386 go build -o ll-analyzer

#windows
GOOS=windows GOARCH=amd64 go build -o ll-analyzer.exe

# linux
GOOS=linux GOARCH=amd64 go build -o ll-analyzer

Usage

Create a directory logs or with number of the incident you are working with. (windows users Avoid the Desktop or another directory with subdirectories).
Download or copy logs files to that directory.
Open a terminal or window command line.
Do a export PATH to the ll-analyzer. On Windows and environment variable should be created.
just execute the tool.

Confog files

config.toml, is the configuration file where other kind of logs can be configured.
console.tpl, is a the template file for console log reports
jdbc.tpl, is a the template file for console log reports

Extensibility

Currently the tools provide analysis for two logs, console and jdbc. But, it can be extended to analyse other logs as well.

Why Go?

Learn a new language
Easily compilation and distribution of the tool between platforms
Use of goroutines and channels (cheap threads)

How to implement a new log analyzer?

1.- Configuration. You define a set of regular expressions.

consolel logs

[[logs]]
logType = "console"
templateFile = "console.tpl"
reportFileName = "console-report.html"
separator = "|"
splitSeparator = "\n"
lineMatchRegex = '((?P<starting>^((STATUS \||ERROR  \||INFO   \|)( jvm 1    \|| wrapper  \|)( main    \|)))(?P<timestamp> \d{4}\/\d{2}\/\d{2} \d{2}:\d{2}:\d{2}.\d{3}))'
fieldsMatchRegex  = [
           ['errorLine','((?P<starting>^((STATUS \||ERROR  \||INFO   \|)( jvm 1    \|| wrapper  \|)( main    \|)))(?P<timestamp> \d{4}\/\d{2}\/\d{2} \d{2}:\d{2}:\d{2}).\d{3} \|).*?ERROR(?P<thread>(\s\[[A-Za-z0-9-:\.\s\[]+\])).*(?P<job>(\[[a-zA-Z_\.]+\]|\([a-zA-Z_\.\-]+\)))(?P<message>\s(.*))'],
           ['exceptionLine','((?P<starting>^((STATUS \||ERROR  \||INFO   \|)( jvm 1    \|| wrapper  \|)( main    \|)))(?P<timestamp> \d{4}\/\d{2}\/\d{2} \d{2}:\d{2}:\d{2}).\d{3} \|).*(?P<exception>\s([a-zA-Z\.]*Exception))(\s|:)(?P<message>(.*))'],
           ['causedby','(?P<causedby>(\sCaused by:.+))'],
           ['causedbyLine','((?P<starting>^((STATUS \||ERROR  \||INFO   \|)( jvm 1    \|| wrapper  \|)( main    \|)))(?P<timestamp> \d{4}\/\d{2}\/\d{2} \d{2}:\d{2}:\d{2}).\d{3} \|).*?(Caused\sby:)(?P<causedby>(\s.*Exception))(\s|:)(?P<message>(.*))'],
]

jdbc logs

[[logs]]
logType = "jdbc"
templateFile = "jdbc.tpl"
reportFileName = "jdbc-report.html"
separator = "|"
splitSeparator = "\n"
lineMatchRegex = '\|(\d{1,10}) ms\|'
fieldsMatchRegex  = [
           ['threadId','^(\d{1,10})\|'],
           ['dataSourceId','^*(master)'],
           ['dateAndTime','^*((\d{2})(\d{2})(\d{2})*-([0-1]\d|2[0-3]):([0-5]\d):([0-5]\d):(\d{3}))'],
           ['executionTime','\|(\d{3,10}) ms\|'],
           ['category','^*(statement)'],
           ['statement','(SELECT|INSERT INTO|UPDATE|WITH|DELETE).*\|'],
           ['sql','\|(INSERT INTO|UPDATE|SELECT|WITH|DELETE)[^?]*$'],
           ['trace','\/\*(.*?)END'],
]

2.- Define the structs that represent the data.

report.go


type FindIssue func(wg *sync.WaitGroup, line string, data *Dataset, rpt *Report) (kv *KeyValue)


type Report struct {
   sync.RWMutex
   ReportType    string
   NumberOfFiles int
   Issues        map[string]interface{}
   Template
   FindIssue
}

type KeyValue struct {
   Key   string
   Value interface{}
}


type JDBCIssue struct {
   Statement         string
   Occurrences       int
   TotalTime         int
   HigherTimeMillis  int
   AverageTimeMillis int
   IssueFileNames    map[string]string
   Sqls              map[string]SqlIssue
}

type SqlIssue struct {
   Sql   string
   Trace []string
}

3.- Implement a type function called FindIssue

jdbc.go


func FindIssueJdbs(wg *sync.WaitGroup, line string, d *Dataset, rpt *Report) (kv *KeyValue) {
   lf := d.LogFile
   mapRegEx := d.MapRegEx

   isAndIssue := isJDBCIssue(mapRegEx, line, lf.Separator)
   if !isAndIssue {
      return nil
   }

   mapStrValues := make(map[string]string)
   for k, v := range mapRegEx {
      if matchedField := tools.MatchStringInLine(line, v, lf.Separator); matchedField != "" {
         mapStrValues[k] = matchedField
      }
   }

   kv = addJDBCIssue(mapStrValues, lf, rpt)

   return kv

}

4.- Register that function

func registerReportFunctions(rpt *Report) (error) {

   errMsg := fmt.Sprintf("there is not 'FindIssue()' function for this report type %s", rpt.ReportType)

   switch rpt.ReportType {

   case "jdbc":
      rpt.FindIssue = FindIssueJdbs

   case "console":
      rpt.FindIssue = FindIssueConsole

   case "thread-dump":
      return errors.New(errMsg)

   default:
      return errors.New(errMsg)
   }

   return nil
}

5.- Create a text template or html template. Exmples are jdbc.tpl and console.tpl

For OO Developers. Is Go OO?