The goal of this project is to provide tools for working with large network traffic datasets and to facilitate research in the traffic classification area. The core functions of the cesnet-datazoo
package are:
- A common API for downloading, configuring, and loading of three public datasets of encrypted network traffic.
- Extensive configuration options for:
- Selection of train, validation, and test periods.
- Selection of application classes and splitting classes between known and unknown.
- Data transformations, such as feature scaling.
- Built on suitable data structures for experiments with large datasets. There are several caching mechanisms to make repeated runs faster, for example, when searching for the best model configuration.
- Datasets are offered in multiple sizes to give users an option to start the experiments at a smaller scale (also faster dataset download, disk space, etc.). The default is the
S
size containing 25 million samples.
🧠🧠See a related project CESNET Models providing pre-trained neural networks for traffic classification. 🧠ðŸ§
📓 📓 Example Jupyter notebooks are included in a separate CESNET Traffic Classification Examples repo. 📓 📓
The following datasets are available in the cesnet-datazoo
package:
Name | CESNET-TLS22 | CESNET-QUIC22 | CESNET-TLS-Year22 |
---|---|---|---|
Protocol | TLS | QUIC | TLS |
Published in | 2022 | 2023 | 2023 |
Collection duration | 2 weeks | 4 weeks | 1 year |
Collection period | 4.10.2021 - 17.10.2021 | 31.10.2022 - 27.11.2022 | 1.1.2022 - 31.12.2022 |
Application count | 191 | 102 | 180 |
Available samples | 141392195 | 153226273 | 507739073 |
Available dataset sizes | XS, S, M, L | XS, S, M, L | XS, S, M, L |
Cite | https://doi.org/10.1016/j.comnet.2022.109467 | https://doi.org/10.1016/j.dib.2023.108888 | |
Zenodo URL | https://zenodo.org/record/7965515 | https://zenodo.org/record/7963302 | |
Related papers | https://doi.org/10.23919/TMA58422.2023.10199052 |
Install the package from pip with:
pip install cesnet-datazoo
or for editable install with:
pip install -e git+https://github.com/CESNET/cesnet-datazoo
from cesnet_datazoo.datasets import CESNET_QUIC22
from cesnet_datazoo.config import DatasetConfig, AppSelection
dataset = CESNET_QUIC22("/datasets/CESNET-QUIC22/", size="XS")
dataset_config = DatasetConfig(
dataset=dataset,
apps_selection=AppSelection.ALL_KNOWN,
train_period_name="W-2022-44",
test_period_name="W-2022-45",
)
dataset.set_dataset_config_and_initialize(dataset_config)
train_dataframe = dataset.get_train_df()
val_dataframe = dataset.get_val_df()
test_dataframe = dataset.get_test_df()
The DatasetConfig
class handles the configuration of datasets, and calling set_dataset_config_and_initialize
initializes train, validation, and test sets with the desired configuration.
Data can be read into Pandas DataFrames as shown here or via PyTorch DataLoaders. See CesnetDataset
reference.
See more examples in the documentation.
- DataZoo: Streamlining Traffic Classification Experiments
Jan Luxemburk and Karel Hynek
CoNEXT Workshop on Explainable and Safety Bounded, Fidelitous, Machine Learning for Networking (SAFE), 2023
This project was supported by the Ministry of the Interior of the Czech Republic, grant No. VJ02010024: Flow-Based Encrypted Traffic Analysis.