Code for the HPCC-2022 paper "FedTADBench: Federated Time-series Anomaly Detection Benchmark"

Note: All the command in the code blocks are based on the root directory of this repository.

Experiments Settings

Time Series Anomaly Detection Methods: DeepSVDD, GDN, LSTM_AE, Tran_AD, USAD
Federated Learning Methods: FedAvg, FedProx, SCAFFOLD, MOON
Datasets: SMD, SMAP, PSM
GPUs:
- GDN: NVIDIA Geforce GTX 1080TI (11GB) GPU
- Others: NVIDIA Geforce RTX 3090 (24GB) GPU
Python Packages:
- GDN: Python 3.7, PyTorch 1.4.0, torch-geometric 1.5.0
- Others: Python 3.8, PyTorch 1.11.0, torch-geometric 2.1.0.post1

Datasets

SMD

directory structure:
- data/datasets/smd/SMD/raw
  - train
  - test
  - test_label
  - list.txt
url: https://github.com/NetManAIOps/OmniAnomaly/tree/master/ServerMachineDataset

SMAP

directory structure:
- data/datasets/smd/SMAP/raw
  - train
  - test
  - test_labels
  - list.txt

PSM

directory structure:
- data/datasets/smd/PSM/raw
  - train
  - test
  - test_labels
  - list.txt
url: https://github.com/eBay/RANSynCoders/tree/main/data

Usage

before running the code, please create directories:

fltsad
- pths
- pths_average
- scores
- scores_average

Experiments Corresponding to Table II

Centralized

# for Time Series Anomaly Detection Method: DeepSVDD
cd algorithms/DeepSVDD
python deepsvdd_exp_[Dataset(smd/smap/psm)].py

# for Time Series Anomaly Detection Method: GDN
cd algorithms/GDN
python gdn_exp_[Dataset(smd/smap/psm)].py

# for Time Series Anomaly Detection Method: LSTM_AE
cd algorithms/DeepSVDD
python lstmae_exp_[Dataset(smd/smap/psm)].py

# for Time Series Anomaly Detection Method: TranAD
cd algorithms/DeepSVDD
python tranad_exp_[Dataset(smd/smap/psm)].py

# for Time Series Anomaly Detection Method: USAD
cd algorithms/DeepSVDD
python usad_exp_[Dataset(smd/smap/psm)].py

Federated

# for Time Series Anomaly Detection Methods: GDN, LSTM_AE, Tran_AD, USAD
python main.py --alg [Federated Learning Method (fedavg/fedprox/scafold/moon)] --tsadalg [Time Series Anomaly Detection Method (gdn/lstm_ae/tran_ad/usad)] --dataset [Dataset Name(smd/smap/psm)]

# for Time Series Anomaly Detection Method: DeepSVDD
python main_svdd.py --alg [Federated Learning Method (fedavg/fedprox/scafold/moon)] --tsadalg deepsvdd --dataset [Dataset Name(smd/smap/psm)]

Experiments Corresponding to Fig. 5.

python main.py --alg [Federated Learning Method (fedavg/fedprox/scafold/moon)] --tsadalg usad --dataset psm --beta [0.1/0.5/5/1000000]

Note: when setting beta >= 10000 in our code, it will be a uniform distribution instead of Dirichlet distribution with beta equals to that beta value.

Experiments Corresponding to Fig. 6.

# for Time Series Anomaly Detection Methods: GDN, LSTM_AE, Tran_AD, USAD
python main_clients_average.py --alg [Federated Learning Method (fedavg/fedprox/scafold/moon)] --tsadalg [Time Series Anomaly Detection Method(gdn/lstm_ae/tran_ad/usad)] --dataset psm

# for Time Series Anomaly Detection Method: DeepSVDD
python main_clients_average_svdd.py --alg [Federated Learning Method (fedavg/fedprox/scafold/moon)] --tsadalg deepsvdd --dataset psm

Results Corresponding to Table III, Table IV, Fig. 4. and Fig. 6.

After running the experiments before, the anomaly scores are stored in fltsad/scores and fltsad/scores_average. Run

python calc_precision_recall_f1_point_adjust.py

to get the Precision, Recall, F1 without/with point adjusting.

Code Reference

DeepSVDD: https://github.com/lukasruff/Deep-SVDD-PyTorch
GDN: https://github.com/d-ailin/GDN
LSTM_AE: https://github.com/astha-chem/mvts-ano-eval
TranAD: https://github.com/imperial-qore/TranAD
USAD: https://github.com/manigalati/usad
SCAFFOLD：https://github.com/Xtra-Computing/NIID-Bench
MOON: https://github.com/QinbinLi/MOON

Contact

If any questions about the code, please contact fanxingliu2020@gmail.com

fanxingliu2020/FedTADBench