/loghub

A large collection of system log datasets for AI-powered log analytics

Loghub

Loghub maintains a collection of system logs, which are freely accessible for research purposes. Some of the logs are production data released from previous studies, while some others are collected from real systems in our lab environment. Wherever possible, the logs are NOT sanitized, anonymized or modified in any way. All these logs amount to over 77GB in total. We thus host only a small sample (2k lines) on Github for each dataset.

🔭 If you use the loghub datasets in your research for publication, please kindly cite the following paper.

How to get the data?

If you are interested in these datasets, please request the raw logs via Zenodo.

Logs currently available:

Software System Description Labeled Time Span #Messages Data Size
Distributed systems
HDFS Hadoop distributed file system log ✔️ 38.7 hours 11,175,629 1.47GB
Hadoop distributed file system log N.A. 71,118,073 16.06GB
Hadoop Hadoop mapreduce job log ✔️ N.A. 394,308 48.61MB
Spark Spark job log N.A. 33,236,604 2.75GB
Zookeeper ZooKeeper service log 26.7 days 74,380 9.95MB
OpenStack OpenStack infrastructure log ✔️ N.A. 207,820 58.61MB
Supercomputers
BGL Blue Gene/L supercomputer log ✔️ 214.7 days 4,747,963 708.76MB
HPC High performance cluster log N.A. 433,489 32.00MB
Thunderbird Thunderbird supercomputer log ✔️ 244 days 211,212,192 29.60GB
Operating systems
Windows Windows event log 226.7 days 114,608,388 26.09GB
Linux Linux system log 263.9 days 25,567 2.25MB
Mac Mac OS log 7.0 days 117,283 16.09MB
Mobile systems
Andriod Andriod framework log N.A. 1,555,005 183.37MB
HealthApp Health app log 10.5 days 253,395 22.44MB
Server applications
Apache Apache web server error log 263.9 days 56,481 4.90MB
OpenSSH OpenSSH server log 28.4 days 655,146 70.02MB
Standalone software
Proxifier Proxifier software log N.A. 21,329 2.42MB

Publications using these datasets

Organizations that download these datasets

🔭 We proudly announce that the loghub datasets have been downloaded 7400+ times by more than 360+ organizations from both industry and academia.

Additional Logs

We have some links to additional log datasets that are related to security research.

Feedback

For any questions or feedback, please post to our issue page.

License

The log datasets are freely available for research purposes.