About "detection adjustment" in the line 339-360 of solver.py
wuhaixu2016 opened this issue · 6 comments
Since some researchers are confused about the "detection adjustment", we provide some clarification here.
(1) Why use "detection adjustment"?
Firstly, I strongly suggest the researchers read the original paper Xu et al., 2018, which has given a comprehensive explanation of this operation.
In our paper, we follow this convention because of the following reasons:
- Fair comparison: As we stated in the Implementation details section of our paper, the adjustment is a widely-used convention in time series anomaly detection. Especially, in the benchmarks that we used in our paper, the previous methods all use the adjustment operation for the evaluation of these benchmarks Shen et al., 2020. Thus, we also adopt the adjustment for model evaluation.
- Real-world meaning: Since one abnormal event will cause a segment of abnormal time points. The adjustment corresponds to the "abnormal event detection" task, which is to evaluate the model performance in detecting the abnormal events from the whole records. This is a very meaningful task for real-world applications. Once we have detected the abnormal event, we can send a worker to check that time segment for security.
In summary, you can view the adjustment as an "evaluation protocol", which is to measure the capability of models in "abnormal event detection".
(2) We have provided a comprehensive and fair comparison in our paper.
- All the baselines that we compared in our paper are also evaluated with this "adjustment". Note that this evaluation is widely used in the previous papers for the experiments on SMD, SWaT, and so on. Thus, the comparison is fair.
- For a comprehensive analysis, we also provide a benchmark for the UCR dataset in Appendix L, which is from KDD Cup. The anomalies in this dataset are mostly recorded only at a single time point. Thus, if you want to obtain the comparison on single-time-point anomaly detection, this dataset can provide some intuitions.
If you still have some questions about the adjustment, welcome to email me and discuss more (whx20@mails.tsinghua.edu.cn).
您好，请问一下，如果要将您的这个成果应用在别的领域的话。您异常检测出来的值和怎么导出来然后从原始数据中过滤掉？
有无学者知道如何解决不知道测试标签的情况下解决该问题嘛
您好，这个调整只是用于计算metric，如果您是想用于部署的话，直接注释掉就可以了 @xiaobiao998
这个代码的性能全靠看了GT后的调整，在现实中没有GT，也就是你注释掉的效果，如果我是reviewer，我会强烈要求去掉这个adjustment，完全是为了好看的不现实步骤。这个不能叫fair compare，因为adjustment有可能利于作者提出的方法，而且这个adjustment现实中不存在，毫无意义。
这个代码的性能全靠看了GT后的调整，在现实中没有GT，也就是你注释掉的效果，如果我是reviewer，我会强烈要求去掉这个adjustment，完全是为了好看的不现实步骤。这个不能叫fair compare，因为adjustment有可能利于作者提出的方法，而且这个adjustment现实中不存在，毫无意义。
烦请仔细阅读上面的说明。为了清晰，我们提供一个中文版本，见下
（1）**公平比较：**自从2018年Xu等论文之后，【大部分工作】全都遵循这一调整，所以我们也使用这个技巧
（2）**实际意义：**您可以考虑这样一个场景，在实际部署中，我们的模型定位出了一个异常点，可以派工人前去查看前后一段时间，所以完全不需要有gt，依然可以实现现实部署。所以，使用调整之后，可以理解为是“基于异常事件”的指标。