/AbuseDetectionSystem_demo

Provide a Deep Learning-Based real-time solution for nursing homes and hospitals for detecting cases of abuse in the elderly population by analyzing security camera frames and performing real-time forecasting using three machine learning models YOLO, DeepSort, ADS

Primary LanguagePython

Abuse Detection System

Project creators

Provide a Deep Learning-Based real-time solution for nursing homes and hospitals for detecting cases of abuse in the elderly population by analyzing security camera frames and performing real-time forecasting using three machine learning models YOLO, DeepSort, ADS

Table of contents

Project highlights

  • Creating a new novel dataset ADS-dataset that contain worldwide videos clips of abuse capture by surveillance cameras in the real-world scenes

  • Design and implemented ML pipeline for video raw data to generate features, data augmentation techniques, and resampled datasets for model training.

  • Build and train a machine learning model[ADS] based on MobileNet-SSD architecture with 3D-CNN and two-stream method [RGB, OPT]. Training and evaluation of the model using AWS-SageMaker and TensorFlow frameworks. Achieved 87% in F1-score on DCSASS Abuse Dataset and 84% on ADS Dataset.

  • Combine all models to an overall system and deploying the system in the Streamlit web app that enables the user to get real-time notification alerts when an abuse event capture by the security camera.

Introduction

This project is defined as research(60%)\development(40%).

  • Research Build and train deep learning models(according to a standard ML approach) to automatically identify abuse event capture by security camera

  • Development Build prototype system ADS(Abuse detection system) for deploying models and test them in a real-time environment

Our Main goal - provide an automated solution for detecting cases of abuse of the elderly.

Data collection

Data collection In order to train deep learning models, the first step is data collection We build data collection pipe and gather abuse video from the web we collect 842 video clips after the filtering process

We work according to a machine learning methodology

  1. search abuse video links online
  2. download the links
  3. convert the video to AVI format with FBS=30sec
  4. cut the video into 5-sec clips
  5. manual extracting from each video 5sec clips [3,4 clips for each video]
  6. create more videos by using 5 data argumentation techniques
  7. split the data to Train, Val, Test as shown in table2

Model architecture

The model architecture is based on mobileNet SSD. And the highlight of this model is utilizing a branch of the optical flow channel to help build a pooling mechanism.

  • Conv3D split into two channels - RGB frame and Optical flows as shown in the figure below.
  • Relu activation is adopted at the end of the RGB channel.
  • Sigmoid activation is adopted at the end of the Optical flow channel.
  • RGB and Optical Flow channels outputs are multiplied together and processed by a temporal max-pooling.
  • Merging Block is composed of basic 3D CNNs, used to process information after self learned temporal pooling.
  • Fully-connected layers generate output.

Model training && Evaluation

Model training

We conduct tow experiment with different optimization algorithm Stochastic gradient descent (SGD) and Adam

Evaluation

  • Full Evaluation report Model evaluation report.pdf

  • We evaluate our proposed model on five Dataset.

    1-ADS data set

    2-DCSASS Abuse Dataset

    3-Automatic violence detection data set

    4- DCSASS Abuse Dataset + Automatic violence detection data set

    5-RWF real world fight dataset

  • F1 score result

res_f1_summary

ADS PIPELINE Demo

First step:

  • We sample from the IP camera 2 frame set each set containing 149 frame set, and then we saves the sampling video in a folder

First step

Second step:

  • Open sampling video folder and run yolo and DeepSort models to predict tracking bounding boxes in the video
  • Bluer each frame set by bounding boxes then save those video clips in a folder

Second step

Third step:

  • We open the video folder implement ads model preprocessing [resize shape, extract optical flow , uniform sampling to 64 frame for predication]

Third step

  • if the ads model identified the frame set as violence, we save the video clip and send it to the user email address

email

System overview

ads overview2

Input-Output examples

EX-1

EX-2

EX-3

EX-4

EX-5

EX-6

Setup

  • First step downlaod yolo_v3 model Yolo_v3 model
  • Add the Yolo_v3.h5 to model data folder
  • Add the Yolo_v3.h5 path to Yolo_v3.py in init function
self.model_path = r'./model_data/yolov3_model.h5'

In ADS_pipeLine.py change to following

  • provide main_folder_output path
  • provide src video input path
main_folder_output = r""
src_video_input = r""
user_email = None
ads_wights_path = r".\Model_to_test\model_json_format\ADS_weights.h5"
ads_model_path = r".\Model_to_test\model_json_format\ADS_model.json"
deep_sort_model_path = ".\deep_sort/mars-small128.pb"

If you want to send an email alert to the user

  • fill in the user_email to the address you want to send
user_email = None
  • go to ADS_pipeline.py in the send_email_alert function
 fromaddr = "<your email account>"
 EMAIL_PASSWORD ="<your email password>"

Reference

@article{RWF-2000,
  author = {Ming Cheng, Kunjing Cai, and Ming Li},
  title={RWF-2000: An Open Large Scale Video Database for Violence Detection},
  year={2019}
}

@article{DeepSort,
  author = { B. A. &. P. D. Wojke},
  title={Simple online and realtime tracking with a deep association metric},
  year={2017}
}

@article{YOLOv3,
  author = {Joseph Redmon },
  title={YOLOv3: An Incremental Improvement},
  year={2018}
}