pierg/anomaly-detection

Anomaly Detection in Event Streams: Leveraging advanced deep learning for anomaly detection in event streams, with metrics evaluation, Slurm support, and Hadoop HDFS validation.

anomaly detection

* (Image generated by ChatGPT symbolizing anomaly detection in event data streams)

Overview

Anomaly detection system using various deep learning models, including Transformer and LSTM architectures, to identify anomalies in event data streams. It benchmarks against traditional models like DeepLog, using precision, recall, and F1 score metrics. The system supports Slurm job management and offers a general framework for modeling event data streams. Validated on the Hadoop HDFS dataset, it demonstrates effectiveness in real-world scenarios.

---

Getting Started

System Requirements:

• Python: version 3.11
• Poetry for dependencies management

Installation

1. Clone the anomaly-detection repository:

$ git clone https://github.com/pierg/anomaly-detection

2. Change to the project directory:

$ cd anomaly-detection

3. Install the dependencies:

$ poetry install

Usage

Run anomaly-detection using the command below:

$ python main.py

---

Repository Structure

└── anomaly-detection/
    ├── README.md
    ├── anomaly-detection
    │   ├── __init__.py
    │   ├── configs
    │   ├── data
    │   ├── main.py
    │   ├── models
    │   ├── optimizers
    │   ├── series
    │   ├── trainers
    │   └── utils
    ├── data
    │   └── hdfs_deeplog
    ├── pyproject.toml
    └── slurm
        ├── cacel.sh
        ├── clean_up.sh
        ├── j_main.sh
        ├── logs.sh
        ├── queue.sh
        └── submit.sh