ArielCyber/OSF-EIMTC

Open-Source Framework for Encrypted Internet and Malicious Traffic Classification.

Framework publish to PyPi

The framework is been uploaded to the pypi for easy use of the framework.

Ariel University

This framework was developed by Ariel Universtiy, you may read more about the framework in our paper: Open-Source Framework for Encrypted Internet and Malicious Traffic Classification

Examples

For usage examples please see the ./examples/ directory for various applications of extracting features from a set of PCAP files or evaluating state-of-the-art deep-learning models.

Installation

Using pip:

python3 -m pip install OSF-EIMTC

Uninstall with:

pythom3 -m pip uninstall OSF-EIMTC

Install tensorflow for deep-learning models: see the official website: https://www.tensorflow.org/install. Generally, you can install tensorflow with pip:

python3 -m pip install tensorflow

But check the offical instructions for more customized options such as for GPU utilization.

For TLS features, TShark is required. While it is possible to install and configure it for Windows usage, the framework is yet to officialy support the system.

sudo apt install tshark

Dependencies

• Platform

  • OS: Linux (Tested with WSL Ubuntu 18.04).
  • Version: Python 3.6.6+ (3.7+ is recommended)

• Python modules

  • NFStream>=6.5.1
  • pandas>=1.1.5
  • numpy>=1.19.5
  • scapy>=2.4.5 (for parsing higher level protocols such as DNS)
  • unittest (for tests)
  • sklearn>=1.1.0

There are some plugins that might require some additional dependancies.

• Optional Python modules

  • pyasn (for ASN info, requires two additional DBs)
  • functools (for caching functions results)
  • click (for CLI)

• For Deep-Learning Models

  • tensorflow>=2.4.0.

• 3rd party tools (that need to be installed)

  • TShark 3.4.8+ (for TLS features). Try install with sudo apt update and then sudo apt install tshark, if you cannot get the 3.0+ version try to install wireshark instead via sudo apt install wireshark.

• 3rd party data files (provided in ./tools directory)

  • pyasn.db (ASN mapping database) - updated one can be downloaded via https://github.com/hadiasghari/pyasn.
  • ip2asn.tsv (ASN context info database) - for updated data download from https://iptoasn.com/.

Run Tests

python3 -m unittest discover EIMTC

Project Structure

Under ./src

└── EIMTC/             # Project's root directory.
    ├── datasets/      # Links and information for PCAP datasets.
    ├── metrics/       # Evaluation metrics for models.
    ├── models/        # State-of-the-art DL models.
    ├── preprocessing/ 
    ├── selection/ 
    ├── plugins/       # NFStream plugins.
    ├── stats/         
    ├── temp/          # Temporary files generated by the system.
    ├── tests/         # System tests.
    │   └── pcaps/     # PCAP files for tests.
    ├── thrid_party/   # 3rd-party tools used by the system.
    └── tools/         # 3rd-party tools used by the system.

Citation

article{BADER2024271,
title = {OSF-EIMTC: An open-source framework for standardized encrypted internet traffic classification},
author = {Ofek Bader and Adi Lichy and Amit Dvir and Ran Dubin and Chen Hajaj},
journal = {Computer Communications},
volume = {213},
pages = {271-284},
year = {2024}
}