/log-parser

Log parsing, and reporting scripts

Primary LanguageShellApache License 2.0Apache-2.0

Log Parsing, Analysis, Correlation, and Reporting Engine written using AWK and Shell script.


There are multiple tools/technologies available in the market that deal (searching, analyzing, monitoring etc.) with machine generated data/logs, but sometimes, you just need some light-weight custom solution to deal with your specific situation and power of Shell scripting helps. I happened to be in such situation when I had to help a project team to identify performance issues, performance deviation, analyse and identify Java memory and Garbage Collection issues, troubleshoot production problem and identify root cause etc. and I ended up writing this solution. I'm sharing this with people, so that they can take it and customize and use as per their need.
If interested, see my blog post - [Log Parsing, Analysis, correlation, and Reporting Engine](https://purnapoudel.blogspot.ca/2018/02/log-parsing-analysing-correlation-reporting-engine.html) for more details.

Project has four parsers namely:


* Utility scripts: * masterLogParser.sh wrapper script for invoking above parsers all sequentially. * usage.sh - utility script - shows the help information.
* Input files: * thresholdValues.csv * WAS_CloneIDs.csv * WASCustomFilter.txt * perfBaseLine.csv

I have written few blog posts about these parsers. See following for details:

  1. Log Parsing, Analysis, correlation, and Reporting Engine - General introduction.
  2. How to Parse WebSphere Application Server Logs for Troubleshooting & Reporting - About WAS parser.
  3. How to Parse Apache access_Log for Troubleshooting & Reporting - about Apache access_log parser.
  4. How to Parse Apache error_log for Troubleshooting & Reporting - about Apache error_log parser.
  5. How to Parse WAS Verbose GC logs for Troubleshooting & Reporting - about Java GC log parser.