how to obtain the performance of some benchmark models reported in the paper?
Opened this issue · 3 comments
Thank you very much for sharing your code. And also the paper is very impressive and useful. Just a question regarding the performance reported in the paper.
In the paper, the performance of ViTAD on the Real-IAD dataset is 84.9 regarding the AUPRO, while in another paper (https://github.com/zhangzjn/ADer?tab=readme-ov-file), this score is 97.3. Obviously, there is a huge performance gap. So are the experimental results of these baseline models are based on your own reimpelmentation?
Thank you.
We directly use the results in the benchmark paper "ADer: A COMPREHENSIVE LIBRARY FOR BENCHMARKING
MULTI-CLASS VISUAL ANOMALY DETECTION", which is exactly the github link you mention.
We use the numbers reported in their paper (Table 5). The numbers posted on their github is actually messed up: the number of AUPRO actally belongs to P-AUROC.
97.3 is obviously too large for AUPRO. Be careful and doubtful with the posted results.
Got you. Thank you very much.
Sorry, I need to reopen this issue. I compared the results reported in the paper ADer: A COMPREHENSIVE LIBRARY FOR BENCHMARKING MULTI-CLASS VISUAL ANOMALY DETECTION with scores reported in your paper. It seems that the scores are still not same. Taking ViTAD on the Real-IAD dataset as an example, the image-level mAU-ROC in your paper 82.3, while in the attached Table 5, it is 82.7. Any reason for this difference?