Source code and supplementary materials for "An Experimental Evaluation of Anomaly Detection in Time Series".
To enable reproductivity, we open source all datasets, algorithms and codes introduced in the paper, and this document produces a guideline of reproduction.
To ease reproducibility, we put it in Google Drive https://drive.google.com/drive/folders/1BJuiWMmBzK2UeiGahVoX2ofqlShELjkN
- java:1.8
- python: 3.7
- modules needed: java in lib, python in requirements.txt
- Section 4.2.1 Varying Datasets
- acc
- Section 4.2.2 Varying Anomaly Rate
- rate
- Section 4.2.3 Varying Data Size
- size
- Section 4.2.4 Varying Data Dimension
- dim
- Section 4.2.5 Varying Anomaly Patterns
- type
- Section 4.3.1 Univariate Methods in Multivariate Datasets
- inter/UMPerDimTest
- Section 4.3.2 Online Methods in Batched Time Series
- inter/SlideTest
- Section 4.3.3 Point Methods in Subsequence Anomalies
- inter/SPLengthTest
- Section 4.3.4 Effect of Metrics and Adjustment
- inter/AdjustPredictTest
- Section 4.3.5 Effect of Threshold
- threshold
- Section 4.3.6 Application
- application
Take the code of drawing Table 4 as an example:
// Set parameters
String[] vars = {"yahoo","twitter"};
boolean[] willOperate = {true, true, true};
String[] algNames = {"NETS", "Stare","SHESD"};
String[] metricNames = {"precision", "recall", "fmeasure"};
...
// NETS
algIndex++;
if (willOperate[algIndex]) {
System.out.println(algNames[algIndex] + " begin");
if (!seriesMulMap.containsKey(dsName)) {
timeSeriesMulDim = fh.readMulDataWithLabel(rawPath);
seriesMulMap.put(dsName, timeSeriesMulDim);
TreeMap<Long, TimePointMulDim> realAnomaly =
DataHandler.findAnomalyPoint(timeSeriesMulDim);
realAnomalyMulMap.put(dsName, realAnomaly);
} else {
timeSeriesMulDim = seriesMulMap.get(dsName);
}
algtime[algIndex][0] = System.currentTimeMillis();
nets = new NETS();
Map<String, Object> netsParams = meta.getDataAlgParam().get(dsName).get(algNames[algIndex]);
nets.init(netsParams, timeSeriesMulDim);
nets.run();
algtime[algIndex][1] = System.currentTimeMillis();
DataHandler.evaluate(
timeSeriesMulDim, realAnomalyMulMap.get(dsName), metrics[index][algIndex]);
}
...
//Write results
fh.writeResults("acc", "uni-point", vars, algNames, metricNames, totaltime, metrics, 1);
Result:
Dataset | NETS | Stare | SHSED | ||||||
---|---|---|---|---|---|---|---|---|---|
Precison | Recall | Fmeasure | Precison | Recall | Fmeasure | Precison | Recall | Fmeasure | |
Yahoo | 0.727 | 1 | 0.842 | 0.429 | 0.375 | 0.400 | `1 | 0.625 | 0.769 |
0.739 | 0.878 | 0.802 | 0.203 | 0.959 | 0.335 | `0.260 | 0.176 | 0.210 |