/ADer

ADer is an open source visual anomaly detection toolbox based on PyTorch, which supports multiple popular AD datasets and approaches.

Primary LanguagePython

  • ADer is an open source visual Anomaly Detection toolbox based on PyTorch, which supports multiple popular AD datasets and approaches.
  • We reproduce popular AD methods under the Multi-class Unsupervised Anomaly Detection (MUAD) by default.
  • We hope it can bring convenience to your research and application.

🐉 News

  • 🔥 We have released several reproduced models, configuration files, and training logs in our benchmark paper 🐲 Paper | Results & CFGs
  • 🔥 COCO-AD and powerful InvAD is released 🐲 Paper | Project | Code
  • 🔥 Plain ViT based ViTAD is released 🐲 Paper | Project | Code
  • 🔥 Real-IAD is released: a new large-scale challenging industrial AD dataset 🐲 Paper | Project | Code

💡 Property

🛠️ Getting Started

Installation

  • Clone this repo:

    git clone https://github.com/zhangzjn/ader.git && cd ader

  • Prepare general experimental environment

    pip3 install timm==0.8.15dev0 mmselfsup pandas transformers openpyxl imgaug numba numpy tensorboard fvcore accimage Ninja
pip3 install mmdet==2.25.3
pip3 install --upgrade protobuf==3.20.1 scikit-image faiss-gpu
pip3 install adeval
pip3 install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 --index-url https://download.pytorch.org/whl/cu118
pip3 install fastprogress geomloss FrEIA mamba_ssm adeval fvcore==0.1.5.post20221221
(or) conda install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 pytorch-cuda=11.8 -c pytorch -c nvidia

Dataset Preparation

Please refer to Datasets Description for preparing visual AD datasets as needed.

  • Real-IAD: A new large-scale challenging industrial AD dataset, containing 30 classes with totally 151,050 images; 2,000 ∼ 5,000 resolution; 0.01% ~ 6.75% defect proportions; 1:1 ~ 1:10 defect ratio.
  • COCO-AD: Large-scale and general-purpose challenging AD-adapted dataset.
  • [MVTec AD](data/README.md/###MVTec AD): Most popular AD dataset.
  • VisA: Popular AD dataset.
  • Uni-Medical: Unified medical AD dataset.
  • (Opt.) [MVTec 3D-AD](data/README.md/###MVTec 3D-AD): Improved 3D version of MVTec AD.
  • (Opt.) Cifar10 & Cifar100: For one-class-train, one-class-test, and unified settings.
  • (Opt.) Tiny-ImageNet: A larger one-class dataset.

Train (Multi-class Unsupervised AD setting by default, MUAD)

  • Check data and model settings for the config file configs/METHOD/METHOD_CFG.py
  • Train with single GPU example: CUDA_VISIBLE_DEVICES=0 python run.py -c configs/METHOD/METHOD_cfg.py -m train
  • Train with multiple GPUs (DDP) in one node:
    • export nproc_per_node=8
    • export nnodes=1
    • export node_rank=0
    • export master_addr=YOUR_MACHINE_ADDRESS
    • export master_port=12315
    • python -m torch.distributed.launch --nproc_per_node=$nproc_per_node --nnodes=$nnodes --node_rank=$node_rank --master_addr=$master_addr --master_port=$master_port --use_env run.py -c configs/METHOD/METHOD_CFG.py -m train.
  • Modify trainer.resume_dir to resume training.
  • Single-class Unsupervised AD (SUAD, not recommend currently)
    1. one GPU-0 for training one grid class in mvtec dataset: CUDA_VISIBLE_DEVICES=0 python3 run.py -c configs/vitad/single_cls/vitad_mvtec_bs16.py -m train data.cls_names=grid trainer.checkpoint=runs/vitad/single_class/vitad_mvtec_bs16/grid
    2. one GPU-0 for training all classes serially in mvtec dataset: python3 runs_single_class.py -d mvtec -c configs/vitad/single_cls/vitad_mvtec_bs16.py -n 1 -m -1 -g 0
    3. $GPU_NUM GPUs for training all classes parallelly in mvtec dataset:: python3 runs_single_class.py -d mvtec -c configs/vitad/single_cls/vitad_mvtec_bs16.py -n $GPU_NUM -m 1
    4. results will be saved in default dir: runs/vitad/single_cls/vitad_mvtec_bs16

Test

  • Modify trainer.resume_dir or model.kwargs['checkpoint_path']
  • Test with single GPU example: CUDA_VISIBLE_DEVICES=0 python run.py -c configs/METHOD/METHOD_cfg.py -m test
  • Test with multiple GPUs (DDP) in one node: python -m torch.distributed.launch --nproc_per_node=$nproc_per_node --nnodes=$nnodes --node_rank=$node_rank --master_addr=$master_addr --master_port=$master_port --use_env run.py -c configs/METHOD/METHOD_CFG.py -m test.

How to Build a Custom Approach

  1. Add a model config cfg_model_MODEL_NAME to configs/__base__
  2. Add configs to configs/MODEL_NAME/CFG.py for training and testing.
  3. Add a model implementation file model/MODEL_NAME.py
  4. Add a trainer implementation file trainer/MODEL_NAME_trainer.py
  5. (Optional) Add specific files to data, loss, optim, etc.

📜 MUAD Results on Popular AD Datasets

  • Red metrics are recommended for comprehensive evaluations.
    Subscripts I, R, and P represent image-level, region-level, and pixel-level, respectively.
  • The following results are derived from the original paper. Since the benchmark re-runs all experiments on the same L40S platform, slight differences in the results are reasonable.

MVTec AD

Method mAU-ROCI mAPI mF1-maxI mAU-PROR mAU-ROCP mAPP mF1-maxP mF1P/.2/.8 mAccP/.2/.8 mIoUP/.2/.8 mIoU-maxP mADI mADP mAD.2/.8 mAD Download
UniAD 97.5 99.1 97.3 90.7 97.0 45.1 50.4 22.4 37.5 13.9 34.2 98.0 64.1 24.6 82.4 log & weight
DiAD 97.2 99.0 96.5 90.7 96.8 52.6 55.5 19.5 40.7 12.0 21.3 97.6 68.3 24.1 84.0 log & weight
ViTAD 98.3 99.4 97.3 91.4 97.7 55.3 58.7 30.9 40.8 20.4 42.6 98.3 70.6 30.7 85.4 log & weight
InvAD 98.9 99.6 98.1 94.1 98.2 57.6 60.1 34.6 46.9 23.0 43.7 98.9 72.0 34.8 86.7 log & weight
InvAD-lite 98.2 99.2 97.2 97.3 55.0 58.1 92.7 32.6 47.1 21.3 41.7 98.2 68.6 33.7 85.4 log & weight

VisA

Method mAU-ROCI mAPI mF1-maxI mAU-PROR mAU-ROCP mAPP mF1-maxP mF1P/.2/.8 mAccP/.2/.8 mIoUP/.2/.8 mIoU-maxP mADI mADP mAD.2/.8 mAD Download
UniAD 88.8 90.8 85.8 85.5 98.3 33.7 39.0 17.9 47.1 10.9 25.7 88.4 57.0 25.3 74.5 log & weight
DiAD 86.8 88.3 85.1 75.2 96.0 26.1 33.0 13.2 46.2 8.0 16.2 86.7 51.7 22.5 70.1 log & weight
ViTAD 90.5 91.7 86.3 85.1 98.2 36.6 41.1 21.6 38.2 13.5 27.6 89.5 58.7 24.4 75.6 log & weight
InvAD 95.5 95.8 92.1 92.5 98.9 43.1 47.0 28.0 45.6 17.9 32.7 94.5 63.0 30.5 80.7 log & weight
InvAD-lite 94.9 95.2 90.8 98.6 40.3 44.3 92.5 25.8 44.2 16.1 30.0 93.6 59.0 28.7 79.5 log & weight

COCO-AD

Method mAU-ROCI mAPI mF1-maxI mAU-PROR mAU-ROCP mAPP mF1-maxP mF1P/.2/.8 mAccP/.2/.8 mIoUP/.2/.8 mIoU-maxP mADI mADP mAD.2/.8 mAD Download
UniAD 56.2 49.0 61.8 31.7 65.4 12.9 19.4 6.6 26.3 3.7 11.1 55.7 32.6 12.2 42.3 log & weight
DiAD 59.0 53.0 63.2 30.8 68.1 20.5 14.2 9.6 31.1 6.1 11.6 58.4 34.2 15.6 44.1 log & weight
ViTAD 69.3 60.4 64.9 41.0 78.3 27.9 31.9 12.4 37.4 7.2 19.8 64.9 46.0 19.0 53.4 log & weight
InvAD 65.9 57.8 64.1 44.9 73.2 19.7 25.4 12.4 37.5 7.1 15.2 62.6 39.4 19.0 50.1 log & weight
InvAD-lite 64.7 56.7 63.5 70.5 18.3 23.4 38.2 11.2 34.2 6.4 13.8 61.6 26.6 17.3 47.9 log & weight

MVTec 3D-AD (RGB)

Method mAU-ROCI mAPI mF1-maxI mAU-PROR mAU-ROCP mAPP mF1-maxP mF1P/.2/.8 mAccP/.2/.8 mIoUP/.2/.8 mIoU-maxP mADI mADP mAD.2/.8 mAD Download
UniAD 78.9 93.4 91.4 88.1 96.5 21.2 28.0 12.2 43.6 7.0 16.8 87.9 48.6 20.9 71.1 log & weight
DiAD 84.6 94.8 95.6 87.8 96.4 25.3 32.3 5.0 71.4 2.6 5.4 91.7 51.3 26.3 73.8 log & weight
ViTAD 79.0 93.1 91.8 91.6 98.2 27.3 33.3 17.2 45.3 10.0 20.5 88.0 52.9 24.1 73.5 log & weight
InvAD 86.1 95.8 93.2 94.7 98.8 37.8 42.5 22.0 50.5 13.2 27.5 91.7 59.7 28.6 78.4 log & weight
InvAD-lite 85.3 95.2 93.0 98.6 37.2 41.4 94.1 21.6 55.3 12.9 26.5 91.2 57.6 30.0 77.8 log & weight

Uni-Medical

Method mAU-ROCI mAPI mF1-maxI mAU-PROR mAU-ROCP mAPP mF1-maxP mF1P/.2/.8 mAccP/.2/.8 mIoUP/.2/.8 mIoU-maxP mADI mADP mAD.2/.8 mAD Download
UniAD 78.5 75.2 76.6 96.4 37.6 40.2 85.0 13.3 37.8 8.0 26.8 76.8 54.3 19.7 69.9 log & weight
DiAD 85.1 84.5 81.2 85.4 95.9 38.0 35.6 19.0 57.6 11.4 25.0 83.6 56.5 29.3 72.2 log & weight
ViTAD 82.2 81.0 80.1 97.2 49.9 49.6 86.1 18.6 36.5 11.7 35.1 81.1 61.9 22.3 75.2 log & weight
InvAD 82.2 79.6 80.6 97.4 47.5 47.1 89.6 21.8 45.2 13.8 33.3 80.8 61.4 26.9 74.9 log & weight
InvAD-lite 79.5 78.3 79.1 96.4 40.1 40.4 85.5 18.3 40.5 11.2 27.6 79.0 55.3 23.3 71.3 log & weight

Real-IAD

Method mAU-ROCI mAPI mF1-maxI mAU-PROR mAU-ROCP mAPP mF1-maxP mF1P/.2/.8 mAccP/.2/.8 mIoUP/.2/.8 mIoU-maxP mADI mADP mAD.2/.8 mAD Download
UniAD 82.9 80.8 74.4 97.4 22.9 30.3 86.4 10.5 35.0 6.0 18.3 79.4 46.5 17.2 67.9 log & weight
DiAD 75.6 66.4 69.9 58.1 88.0 2.9 7.1 2.9 41.9 1.5 3.7 70.6 32.7 15.4 52.6 log & weight
ViTAD 82.7 80.1 73.7 97.3 24.2 32.3 83.9 13.4 27.3 7.7 19.6 78.8 46.8 16.1 67.8 log & weight
InvAD 89.0 86.4 79.6 98.4 30.7 37.6 91.9 17.7 36.1 10.4 23.5 85.0 53.4 21.4 73.4 log & weight
InvAD-lite 87.2 85.1 77.8 98.1 31.6 37.9 91.6 17.6 36.8 10.3 23.7 83.3 53.7 21.6 72.7 log & weight

Citation

If you use this toolbox or benchmark in your research, please cite our related works.

@article{ader,
  title={ADer: A Comprehensive Benchmark for Multi-class Visual Anomaly Detection},
  author={Jiangning Zhang and Haoyang He and Zhenye Gan and Qingdong He and Yuxuan Cai and Zhucun Xue and Yabiao Wang and Chengjie Wang and Lei Xie and Yong Liu},
  journal={arXiv preprint arXiv:2406.03262},
  year={2024}
}

@inproceedings{realiad,
  title={Real-IAD: A Real-World Multi-View Dataset for Benchmarking Versatile Industrial Anomaly Detection},
  author={Wang, Chengjie and Zhu, Wenbing and Gao, Bin-Bin and Gan, Zhenye and Zhang, Jianning and Gu, Zhihao and Qian, Shuguang and Chen, Mingang and Ma, Lizhuang},
  booktitle={CVPR},
  year={2024}
}

@article{vitad,
  title={Exploring Plain ViT Reconstruction for Multi-class Unsupervised Anomaly Detection},
  author={Zhang, Jiangning and Chen, Xuhai and Wang, Yabiao and Wang, Chengjie and Liu, Yong and Li, Xiangtai and Yang, Ming-Hsuan and Tao, Dacheng},
  journal={arXiv preprint arXiv:2312.07495},
  year={2023}
}

@article{invad,
  title={Learning Feature Inversion for Multi-class Anomaly Detection under General-purpose COCO-AD Benchmark},
  author={Jiangning Zhang and Chengjie Wang and Xiangtai Li and Guanzhong Tian and Zhucun Xue and Yong Liu and Guansong Pang and Dacheng Tao},
  journal={arXiv preprint arXiv:2404.10760},
  year={2024}
}

@article{mambaad,
  title={MambaAD: Exploring State Space Models for Multi-class Unsupervised Anomaly Detection},
  author={He, Haoyang and Bai, Yuhu and Zhang, Jiangning and He, Qingdong and Chen, Hongxu and Gan, Zhenye and Wang, Chengjie and Li, Xiangtai and Tian, Guanzhong and Xie, Lei},
  year={2024}
}