/fire-detection-from-images

Detect fire in images using neural nets

Primary LanguageJupyter NotebookMIT LicenseMIT

fire-detection-from-images

The purpose of this repo is to demonstrate a fire detection neural net model. In use this model will place a bounding box around any fire in an image.

Best results

Object detection: After experimenting with various model architectures I settled on Yolov5 pytorch model (see pytorch/object-detection/yolov5/experiment1/best.pt). After a few hours of experimentation I generated a model of mAP@.5 of 0.657, Precision of 0.6, Recall of 0.7, trained on 1155 images (337 base images + augmentation).

Classification: I have yet to train my own model, but 95% accuracy is reported using ResNet50

Segmentation: requires annotation

Motivation and challenges

Traditional smoke detectors work by detecting the physical presence of smoke particles. However they are prone to false detections (e.g. from toasters) and do not localise the fire particularly well. In these situations a camera solution could complement a traditional detector, in order to improve response times or to provide additional metrics such as the size and location of a fire. With the location and nature of the fire identified, an automated intervention may be possible, e.g. via a sprinkler system or drone. Also data can be sent to fire services to provide otherwise non-existent situational awareness. Particular locations I am interested in are: kitchens & living rooms, garages and outbuildings, and areas where fires might already be present but spreading outside a desired zone e.g. fire pit.

There are a couple of significant challenges & open questions:

  • For fast edge model what is 'best' architecture? Yolo3 is very popular for commecrial applications and can be implemented in keras or pytorch, baseline Yolov5 as it is currently SOTA and has deployment guide to Jetson.
  • Can the architecture be optimised since we are detecting only a single class?
  • Baseline object detection, but is there benefit to classifier or segmentation? Obj models train on mAP and Recall metrics but for our application bounding box accuracy may not be top priority? However classification models work best on a nice shot containing only the target object but in real life fire scenarios the scene will not be as simple as this scenario.
  • Tensorflow + google ecosystem or Pytorch + NVIDIA/MS? Tensorflow suffers from tf1 legacy
  • Is a single 'super' model preferable, or several specialised models? Typical categories of fire include candle flame, indoor/outdoor, vehicle
  • Gathering or locating a comprehensive, representative and balanced training dataset
  • Handling different viewpoints, different camera manufacturers and settings, and different ambient lighting conditions.
  • Since fires are so bright they can often wash out images and cause other optical disturbances, how can this be compensated for?
  • Since we expect the model will have limitations, how do we make the model results interpretable?
  • Fires can be a very wide range of sizes, from an candle flame to engulfing an entire forest - is this a small object & large object problem? Splitting dataset by fire class and training models for each class may give better results? Treat as a semantic segmentation problemn (requires reannotating dataset)?

Ideas:

  • Preprocessing images, e.g. to remove background or apply filters
  • Classifying short sequences of video, since the movement of fire is quite characteristic
  • Simulated data, identify any software which can generate realistic fires and add to existing datasets
  • Augmentations to simulate effect of different cameras and exposure settings
  • Identify any relevant guidance/legislation on required accuracy of fire detection techniques
  • Combine RGB + thermal for suppressing false positives? e.g. using https://openmv.io/blogs/news/introducing-the-openmv-cam-pure-thermal or cheaper grideye or melexsis

Approach & Tooling

  • Frames will be fed through neural net. On positive detection of fire metrics are extracted. Ignore smoke for MVP. Try various architectures & parameters to establish a 'good' baseline model.
  • Develop a lower accuracy but fast model targeted at RPi and mobile, and a high accuracy model targeted at GPU devices like Jetson. Yolo present both options, yolo4 lite for mobile and yolo5 for GPU. Alternatively there is mobilenet and tf-object-detection-api. Higher accuracy GPU model is priority.
  • Use Google Colab for training

Articles & repos

Datasets

Fire safety references

  • Locate reference covering the different kinds of fires in the home, common scenarios & interventions
  • Safety/accuracy standards for fire detectors, including ROC characteristics

Fires in the home

  • Common causes including cigarettes left smouldering, candles, electrical failures, chip pan fires
  • A large number of factors affect the nature of the fire, primarily the fuel and oxygenation, but also where the fire is, middle of the room/against a wall, thermal capacity of a room, the walls, ambient temperature, humidity, contaminants on the material (dust, oil based products, emollients etc)
  • To put out a fire a number of retardants are considered - water (not on electrical or chip pan), foam, CO2, dry powder
  • In electrical fires the electricity supply should first be isolated
  • Reducing ventillation, e.g. by closing doors, will limit fire
  • Smoke itself is a strong indicator of the nature of the fire
  • Read https://en.m.wikipedia.org/wiki/Fire_triangle and https://en.m.wikipedia.org/wiki/Combustion

Edge deployment

Our end goal of deployment to an edge device (RPi, jetson nano, android or ios) will influence decisions about architecture and other tradeoffs.

Cloud deployment

We want a solution that could also be deployed to the cloud, with minimal changes vs the edge deployment. A couple of options:

Image preprocessing and augmentation

Roboflow allows up to 3 types of augmentation per dataset, in addition to basic cropping. If we want to experiment with more augmentations we can checkout https://imgaug.readthedocs.io/en/latest/

ML metrics

  • Precision is the accuracy of the predictions, calculated as precision = TP/(TP+FP) or "what % of predictions are correct?"
  • Recall is the true positive rate (TPR), calculated as recall = TP/(TP+FN) or "what % of true positives does the model capture?"
  • The F1 score (also called the F score or the F measure) is the harmonic mean of precision and recall, calculated as F1 = 2*(precision * recall)/(precision + recall). It conveys the balance between the precision and the recall. Ref
  • The false positive rate (FPR), calculated as FPR = FP/(FP+TN) is often plotted against recall/TPR in an ROC curve which shows how the TPR/FPR tradeoff varies with classification threshold. Lowering the classification threshold returns more true positives, but also more false positives
  • mAP, IoU, precision and recall are all explained well here and here
  • IceVision returns the COCOMetric, specifically the AP at IoU=.50:.05:.95 (primary challenge metric), from here, typically referred to as the "mean average precision" (mAP)
  • mAP@0.5: the mean Average Precision or correctness of each label taking into account all labels. @0.5 sets a threshold for how much of the predicted bounding box overlaps the original annotation, i.e. "50% overlap"

Comments

  • Firenet is a VERY common name for model, do not use

Discussion

Demo

The best performing model can be used by running the demo app that created with Gradio. Note you must have the yolov5 repo cloned locally (git clone https://github.com/ultralytics/yolov5) and the path to it

  • python3 -m venv venv
  • source venv/bin/activate
  • pip3 install -r requirements.txt
  • python3 demo.py
  • You are prompted to navigate to http://127.0.0.1:7860/