Generalized Video Anomaly Event Detection: Systematic Taxonomy and Comparison of Deep Models

This is the official repository for our paper entitled “Generalized Video Anomaly Event Detection: Systematic Taxonomy and Comparison of Deep Models” accepted to ACM Computing Surveys. We summarize existing deep learning-based video anomaly event detection (VAED) methods and classify these deep models into four categories: unsupervised, weakly supervised, supervised, and fully unsupervised according to the supervised signal for training. In addition, we collate the public VAED datasets and available codes.

Paper (ACM CSUR Published Version): https://dl.acm.org/doi/abs/10.1145/3645101
Introduction in Chinese (中文版简介): https://mp.weixin.qq.com/s/s8zKuLYVV0k5Z_4fGT2-QQ

Overview

Datasets

Dataset	#Videos	#Normal	#Abnormal	#Scenes	#Anomalies
UMN		6,165	1,576	3	11
Subway Entrance		132,138	12,112	1	51
Subway Exit		60,410	4,491	1	14
Street Scene$^{*}$	81	159,341	43,916	205	17
CUHK Avenue	37	26,832	3,820	1	77
ShanghaiTech	437	300,308	17,090	13	158
UCSD Ped1	70	9,995	4,005	1	61
UCSD Ped2	29	2,924	1,636	1	21
UCF-Crime	1,900				950
ShanghaiTech Weakly$^{**}$	437
XD-Violance	4,754
Ubnormal$^{***}$	543	147,887	89,015	29	660
ADOC			97,030	1	721

$^{*}$ Following previous works, we set the frame rate to 15 fps.

$^{***}$ This dataset is reorganized from ShanghaiTech, so we provide the reorganized file list here.

$^{***}$ This datatset include a validation set that contains 64 videos.

Inference Speed

GVAED typically employs Average Inference Speed (AIS) as a metric for visually gauging the model's overhead cost. Comparisons across reported figures in the existing literature are often challenging due to variations in experimental environments and computational platforms. Recent advancements in GVAED research, such as object-level methods and weakly-supervised schemes, frequently involve intricate data preprocessing and the utilization of pre-trained models. Examples include foreground object detection, optical flow estimation, and spatial-temporal feature extraction using well-trained 3D convolutional networks. It remains unclear whether the computational cost and processing time associated with these aspects are factored into the overhead cost of the proposed model. Consequently, reporting inference speed is not a widespread practice, and the limited works providing such results often lack a comprehensive description of the experimental setup. Nevertheless, diligent efforts were made to aggregate AIS data from existing studies, aiming to offer an insightful overview of the trajectory in lightweight GVAED research. Acknowledging the influence of image resolution on model inference speed, we adhered to the approach outlined by Ramachandra et al., summarizing the data while concurrently documenting the datasets used for model testing. The results are presented below:

Year	Method	AIS (FPS)	Dataset
2010	ADCS	0.4	UCSD Ped2
2011	VParsing	0.13	UCSD Ped1
2013	Avenue	150	CUHK Avenue
2013	SR	0.26	UCSD Ped1
2014	ADL	1.25	UCSD Ped2
2015	RTAD	200	UCSD Ped 1 & Ped2, UMN
2015	STVP	1	UCSD Ped 1 & Ped2
2015	HFR	2	UCSD Ped1
2017	DAF	20	CUHK Avenue
2017	Deep-cascade	130	UCSD Ped 1 & Ped2, UMN
2017	ST-AE	143	CUHK Avenue, Subway, UCSD Ped 1 & Ped2
2017	stacked-RNN	50	UCSD Ped2
2018	Deep-anomaly	370	UCSD Ped2
2018	FFP	25	CUHK Avenue
2019	NNC	24	CUHK Avenue, Subway, UMN
2019	OC-AE	11	CUHK Avenue, UCSD Ped2, SHanghaitch, UMN
2019	mem-AE	38	UCSD Ped2
2019	AnoPCN	10	UCSD Ped2, CUHK Avenue, ShanghaiTech
2020	Clustering	32	UCSD Ped2
2020	MNAD	67	UCSD Ped2
2023	HN-MUM	34	UCSD Ped2
2023	CRC	46	CUHK Avenue

We recommend readers to reproduce the existing methods on their own particular platforms using the publicly available code that we have collected in Section 3, comparing the computational overheads of the various types of methods on a fair measurement benchmark.

Tools

anomalib: An anomaly detection library comprising state-of-the-art algorithms and features such as experiment management, hyper-parameter optimization, and edge inference. Project Page.

PyAnomaly: A PyTorch toolbox for video anomaly detection. Paper, Project Page.

Citation

If you find our work useful, please cite our paper:

@article{liu2024generalized,
  title={Generalized video anomaly event detection: Systematic taxonomy and comparison of deep models},
  author={Liu, Yang and Yang, Dingkang and Wang, Yan and Liu, Jing and Liu, Jun and Boukerche, Azzedine and Sun, Peng and Song, Liang},
  journal={ACM Computing Surveys},
  year={2024},
  publisher={Association for Computing Machinery}
}