Awesome Log Analysis

A curated list of awesome publications and researchers on log analysis, anomaly detection, fault localization, and AIOps.

Awesome Log Analysis

Researchers

China (& HK SAR)
Michael R. Lyu, CUHK	Dongmei Zhang, Microsoft	Pengfei Chen, SYSU	Dan Pei, Tsinghua
Pinjia He, CUHK-Shenzhen
USA
Yuanyuan Zhou, UCSD	Tao Xie, UIUC	Dawson Engler, Stanford	Ben Liblit, Wisconsin–Madison
Canada
Ding Yuan, Toronto University	Ahmed E. Hassan, Queen's University	Weiyi Shang, Concordia University	Zhen Ming (Jack) Jiang, York University
Wahab Hamou-Lhadj, Concordia University
UK

Europe

Australia
Ingo Weber, CSIRO

Conferences and Journals

Logs are a type of valuable data generated from many sources such as software, systems, networks, devices, etc. They have also been used for a number of tasks related to reliability, security, performance, and energy. Therefore, the research of log analysis has attracted interests from different research areas.

System area
- Conferences: OSDI | SOSP | ATC | ICDCS
- Journals: TC | TOCS | TPDS
Cloud computing area
- Conferences: SoCC | CLOUD
- Journals: TCC
Networking area
- Conferences: NSDI | INFOCOMM
- Journals: TON
Software engineering area
- Conferences: ICSE | FSE | ASE
- Journals: TSE | TOSEM
Reliability area
- Conferences: DSN | ISSRE | SRDS
- Journals: TDSC | TR
Security area
- Conferences: CCS | DSN
- Journals: TDSC
AI and Bigdata area
- Conferences: KDD | CIKM | ICDM | BigData
- Journals: TKDE | TBD
Industrial conferences
- SREcon | GOPS

Datasets

Loghub

Papers

Log Mining

Anomaly Detection

[OSDI 2016] Non-intrusive performance profiling for entire software stacks based on the flow reconstruction principle
[FSE 2018] Using finite-state models for log differencing
[ICSE 2016] Behavioral log analysis with statistical guarantees
[FSE 2011] Leveraging existing instrumentation to automatically infer invariant-constrained models
[KDD 2010] Mining program workflow from interleaved traces
[ICSE 2014] Inferring models of concurrent systems from logs of their behavior with CSight
[ASE 2019] Statistical log differencing
[SOSP 2009] Detecting Large-Scale System Problems by Mining Console Logs
[IPOM 2003] A data clustering algorithm for mining patterns from event logs
[FSE 2018] Identifying impactful service system problems via log analysis
[ICSE 2016] Log clustering based problem identification for online service systems
[ICDM 2007] Failure prediction in ibm bluegene/l event logs
[IEICE Transactions on Communications 2018] Proactive failure detection learning generation patterns of large-scale network logs
[ISSRE 2015] Experience report: Anomaly detection of cloud application operations using log and cloud metric correlation analysis
[USENIX ATC 2010] Mining Invariants from Console Logs for System Problem Detection
[ICSE 2013] Assisting developers of big data analytics applications when deploying on hadoop clouds
[ICDM 2009] Online system problem detection by mining patterns of console logs
[ISSRE 2017] Experience report: Log-based behavioral differencing
[KDD 2016] Anomaly detection using program control flow graph mining from execution logs
[ICDM 2009] Execution Anomaly Detection in Distributed Systems through Unstructured Log Analysis
[ASPLOS 2016] Cloudseer: Workflow monitoring of cloud infrastructures via interleaved logs
[KDD 2005] Dynamic syslog mining for network failure monitoring
[ISSRE 2016] Experience report: System log analysis for anomaly detection
[CCS 2017] Deeplog: Anomaly detection and diagnosis from system logs through deep learning
[FSE 2019] Robust log-based anomaly detection on unstable log data
[IJCAI 2019] LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs
[ICCCN 2020] Semantic-aware Representation Framework for Online Log Analysis
[TCCN 2020] An Intelligent Anomaly Detection Scheme for Micro-services Architectures with Temporal and Spatial Data Analysis
[ISSRE 2020] [Cross-System Log Anomaly Detection for Software Systems (to appear)]
[Information Systems Frontiers 2020] LogGAN: a Log-level Generative Adversarial Network for Anomaly Detection using Permutation Event Modeling
[DASC/PiCom/DataCom/CyberSciTech 2018] Detecting anomaly in big data system logs using convolutional neural network
[CCS 2019] Log2vec: A Heterogeneous Graph Embedding Based Approach for Detecting Cyber Threats within Enterprise
[MLCS 2018] Recurrent Neural Network Attention Mechanisms for Interpretable System Log Anomaly Detection

Failure Prediction

Failure Diagnosis

[ICSE 2019] An empirical study on leveraging logs for debugging production failures
[ASPLOS 2016] SherLog: error diagnosis by connecting clues from run-time logs
[ISSTA 2009] AVA:automated interpretation of dynamically detected anomalies
[IC2E 2016] LOGAN: Problem diagnosis in the cloud using log-based reference models
[ICWS 2017] An approach for anomaly diagnosis based on hybrid graph model with logs for distributed services
[Cloud 2017] Logsed: Anomaly diagnosis through mining time-weighted control flow graph in logs
[FSE 2018] CloudRaid: hunting concurrency bugs in the cloud via log-mining
[TPDS 2013] Toward fine-grained, unsupervised, scalable performance diagnosis for production cloud computing systems
[CLUSTER 2014] Digging deeper into cluster system logs for failure prediction and root cause diagnosis
[ASPLOS 2014] Comprehending performance from real-world execution traces: A device-driver case
[ICWS 2017] Log-based abnormal task detection and root cause analysis for spark
[EDCC 2015] Insights into the diagnosis of system failures from cluster message logs
[HPC 2010] Diagnosing the root-causes of failures from cluster log files
[ASE 2019] SCMiner: localizing system-level concurrency faults from large system call traces
[NSDI 2012] Structured comparative analysis of systems logs to diag- nose performance problems
[ICSE 2013] Assisting developers of big data analytics applications when deploying on hadoop clouds
[TPDS 2016] Failure diagnosis for distributed systems using targeted fault injection
[ICSE 2017] What causes my test alarm? Automatic cause analysis for test alarms in system and integration testing
[GLOBECOM 2018] Root-Cause Diagnosis Using Logs Generated by User Actions
[ICSE 2019] Mining Historical Issue Repositories to Heal Large-Scale Online Service Systems
[CLOUD 2019] An Approach to Cloud Execution Failure Diagnosis Based on Exception Logs in OpenStack
[FAST 2009] Understanding customer problem troubleshooting from storage system logs
[DSN 2013] Reading between the lines of failure logs: Understanding how HPC systems fail
[DSN 2014] What logs should you look at when an application fails? insights from an industrial case study
[TSE 2018] Fault analysis and debugging of microservice systems: Industrial survey, benchmark system, and empirical study
[FSE 2019] How bad can a bug get? an empirical analysis of software failures in the OpenStack cloud computing platform

Others

License

This repo is under the MIT license.

JasonHans/awesome-log-analysis