There are multiple tools/technologies available in the market that deal (searching, analyzing, monitoring etc.) with machine generated data/logs, but sometimes, you just need some light-weight custom solution to deal with your specific situation and power of Shell scripting helps. I happened to be in such situation when I had to help a project team to identify performance issues, performance deviation, analyse and identify Java memory and Garbage Collection issues, troubleshoot production problem and identify root cause etc. and I ended up writing this solution. I'm sharing this with people, so that they can take it and customize and use as per their need.
If interested, see my blog post - [Log Parsing, Analysis, correlation, and Reporting Engine](https://purnapoudel.blogspot.ca/2018/02/log-parsing-analysing-correlation-reporting-engine.html) for more details.
Project has four parsers namely:
- websphereLogParser.sh for parsing, analyzing and reporting WebSphere Application Server (WAS) SystemOut.log
- webAccessLogParser.sh for parsing, analyzing and reporting Apache/IBM HTTP Server (IHS) access_log
- webErrorLogParser.sh for parsing, analyzing and reporting Apache/IBM HTTP Server (IHS) error_log
- javaGCStatsParser.sh for parsing, analyzing and reporting Java verbose Garbage Collection (GC) log.
* Utility scripts: * masterLogParser.sh wrapper script for invoking above parsers all sequentially. * usage.sh - utility script - shows the help information.
* Input files: * thresholdValues.csv * WAS_CloneIDs.csv * WASCustomFilter.txt * perfBaseLine.csv
I have written few blog posts about these parsers. See following for details:
- Log Parsing, Analysis, correlation, and Reporting Engine - General introduction.
- How to Parse WebSphere Application Server Logs for Troubleshooting & Reporting - About WAS parser.
- How to Parse Apache access_Log for Troubleshooting & Reporting - about Apache access_log parser.
- How to Parse Apache error_log for Troubleshooting & Reporting - about Apache error_log parser.
- How to Parse WAS Verbose GC logs for Troubleshooting & Reporting - about Java GC log parser.