DeepLog

This is the achievement of core DeepLog for research aim.

Function:

It is the thought on how to use the a series of modules to pre-process the raw system logs to key logs and report the potential malicious logs.

For the dataset, I have given some examples and you can put your own data into that folder.

Instructions (In Deeplog_demo folder):

1. Source data:

When the data format is in csv, we need translate them into txt files and split them into batches.

python3 csv_txt_trans.py

You will get notice on inputing the source location and output location.

2. Data analysis:

we use the logparser tool to transform the source txt log files into structured csv files under a folder, the folder is named by the start and end time. (Find the Lenma_demo under the logparser/logparser/demo)

(use Lenma_demo.py with python2) ---> The python3 version is not provided here. You need to set the locations first:

input_dir = '../../Dataset/Linux/Clear/'   # set the location to yours
output_dir = '../../Dataset/Linux/Clear_Separate_Structured_Logs/'    # set the location to yours

Then you can execute the demo file with python 2.x:

python Lenma_demo.py

In the stage, we calculate the EventTemplate for every log.

3. Variable Selection:

The log_value_vector.py will be used to generate the csv file, which will be used to implement the anomaly detection later.

(and has been integrated into models already in demo)

4. Model detection:

Basiclly, we have two modules for DeepLog

Whereas, before implementing the modules, we will first see whether there is obvious malicious logs, we will report them first.
After that, we will first implement execution path anomaly detection with Execution_Path_Anomaly.py
Finally, we will implement parameter values anomaly detection with Parameter_value_performance_anomaly.py
As a plus, there is the ML model using PCA in loglizer.

python3 PCA_demo_without_labels.py --p1 xxx --p2 xxx

Addition:

If you want to implement the raw version, please run the following command before you go:

pip install -r requirement.txt

Statement:

The model is based on off-line work, the online real-time detection is not available.
The loglizer and logparser are open source tools, author's rights are reserved.
I enriched the two tools in the project, notice the differences from the original version.

References：

1.Execution Anomaly Detection in Distributed Systems through Unstructured Log Analysis

2.DeepLog: Anomaly Detection and Diagnosis from System Logs

bogahariprasad/DeepLog-1