Automotive Ethernet IDS Evaluation Framework

Introduction
Getting started
Folders structure
Citting us

Introduction

This repository contains some of the efforts of the CIn-UFPE Connected Systems Lab on developing new Intrusion Detection Systems (IDSs) for Automotive Ethernet networks.

The main idea of this repository is to contain the code of our work and also the reproduction code of other related work to automotive Ethernet intrusion detection. Therefore, the scripts were constructed to be easily extensible to include new datasets and IDSs that are based on Machine Learning and Deep Learning techniques.

This repository currently contains the code to reproduce the following work:

Getting started

The first step is to prepare the python virtual environment with the necessary libraries. We have prepared a Makefile to help with this setup, to prepare the environment, simply run:

make bootstrap

Once you have the libraries properly installed in the virtual environment, the next step is to activate the virtual environment. To do so, run the following command:

source venv/bin/activate

Once the virutal environment is properly activated, we can move on to use our entrypoint scripts. We have the following entrypoint scripts:

execute_feature_generator.py
execute_model_train_validation.py
execute_model_test.py

Each script uses as input a json configuration file that contain the information necessary for the script execution. The script output depends on the script, but it can vary from data in different formats and trained models to be further used in other scenarios.

These scripts should be ideally executed sequentially with the output of the previous scripts. They were kept separated to ease the experimentation of different kind of parameters that do not require changes in the other scripts.

The overall interaction between the scripts is presented in the following image:

In the following subsections we will present a simple use of each script.

Executing the feature generator script

The main purpose of the feature generator script is to transform raw input data into meaningful features according to the feature generation specification. To execute the feature generator script with a sample input file called feat_gen_config.json, simply run:

python3 execute_feature_generator.py --feat_gen_config feat_gen_config.json

The content of feat_gen_config.json could be:

feat_gen_config.json
{
    "feature_generator": "CNNIDSFeatureGenerator",
    "config": {
        "window_size": 44,
        "window_slide": 1,
        "number_of_bytes": 58,
        "multiclass": false,
        "labeling_schema": "AVTP_Intrusion_dataset",
        "dataset": "AVTP_Intrusion_dataset",
        "suffix": "test",
        "sum_x": true
    },
    "paths": {
        "injected_only_frame_path": "/path/to/file",
        "injected_data_paths" : ["/path/to/file1", "path/to/file2"],
        "output_path": "/output/path"
    }

The output artifacts will be placed in the specific output path.

Executing the model training and validation script

To execute the model training and validation script with a config file named train_val_config.json, simply run:

python3 execute_model_train_validation.py --model_train_valid_config train_val_config.json

A sample content of the train_val_config.json:

train_val_config.json
{
    "feat_gen": {
        "feature_generator": "CNNIDSFeatureGenerator",
        "config": {
            "window_size": 44,
            "window_slide": 1,
            "number_of_bytes": 58,
            "multiclass": false,
            "labeling_schema": "AVTP_Intrusion_dataset",
            "dataset": "AVTP_Intrusion_dataset"
        },
        "load_paths": {
            "X_path": "/path/to/train/x/data",
            "y_path": "/path/to/train/y/data"
        }
    },
    "model_specs": {
        "framework": "pytorch",
        "model": "CNNIDS",
        "model_name": "CNNIDS",
        "criterion": "binary-cross-entropy",
        "hyperparameters": {
            "batch_size": 64,
            "learning_rate": 0.001,
            "num_epochs": 1,
            "early_stopping_patience": 5,
            "num_outputs": 1
        },
        "paths": {
            "metrics_output_path":"/path/to/metric/output",
            "models_output_path":"/path/to/model/output"
        }
    }

Executing the model test script

Finally, to execute the model test script with a config named test.config, simply run:

python3 execute_model_test.py --model_test_config test_config.json

A sample content of the test_config.json:

test_config.json
{
    "feat_gen": {
        "feature_generator": "CNNIDSFeatureGenerator",
        "config": {
            "window_size": 44,
            "window_slide": 1,
            "number_of_bytes": 58,
            "multiclass": false,
            "labeling_schema": "AVTP_Intrusion_dataset",
            "dataset": "AVTP_Intrusion_dataset"
        },
        "load_paths": {
            "X_path": "/path/to/train/x/data",
            "y_path": "/path/to/train/y/data"
        }
    },
    "model_specs": {
        "framework": "pytorch",
        "model": "CNNIDS",
        "model_name": "CNNIDS",
        "criterion": "binary-cross-entropy",
        "hyperparameters": {
            "batch_size": 64,
            "learning_rate": 0.001,
            "num_epochs": 1,
            "early_stopping_patience": 5,
            "num_outputs": 6
        },
        "paths": {
            "metrics_output_path":"/path/to/metric/output",
            "models_output_path":"/path/to/model/output"
        },
        "presaved_paths": {
            "0": "/path/to/model/trained/in/fold/0",
            "1": "/path/to/model/trained/in/fold/1",
            "2": "/path/to/model/trained/in/fold/2",
            "3": "/path/to/model/trained/in/fold/3",
            "4": "/path/to/model/trained/in/fold/4"
        }
    }
}

Folder structure

This repository is organized in the following folder structure:

config_jsons/
- feat_generator/
  - AVTP_CNNIDS_train.json
  - ...
- model_test/
  - TOW_MultiStage_test.json
  - ...
- model_train_validate/
  - AVTP_CNNIDS_train.json
  - ... ...
custom_metrics/
feature_generator/
- abstract_feature_generator.py
- cnn_ids_feature_generator.py
- ...
model_train_validation/
- abstract_model_train_validate.py
- pytorch_model_train_validate.py
- ...
model_test/
- abstract_model_test.py
- pytorch_model_test.py
- ...
models/
- conv_net_ids.py
- multiclass_conv_net_ids.py
- pruned_conv_net_ids.py
- sklearn_classifier.py
- ...
execute_feature_generator.py
execute_model_train_validation.py
execute_model_test.py
Makefile
README.md
requirements.txt
run_framework.sh

Citing us

If you use the code in this repository in your research, please cite our original work:

Marques da Luz, Luigi F. and Freitas de Araujo-Filho, Paulo and Campelo, Divanilson R., Multi-Stage Deep Learning-Based Intrusion Detection System for Automotive Ethernet Networks. Available at SSRN: https://ssrn.com/abstract=4658006 or http://dx.doi.org/10.2139/ssrn.4658006

goon-13/automotive-ids-evaluation-framework