/flutter-realtime-detection-main

Primary LanguageDart

Flutter realtime object detection with Tensorflow Lite

Flutter realtime object detection with Tensorflow Lite

💡 Demo

Info

An app made with Flutter and TensorFlow Lite for realtime object detection using model YOLO, SSD, MobileNet, PoseNet.

⭐ Features

  • Realtime object detection on the live camera

  • Using Model: YOLOv2-Tiny, SSDMobileNet, MobileNet, PoseNet

  • Save image has been detected

  • MVVM architecture


🚀  Installation

  1. Install Packages
camera: get the streaming image buffers
https://pub.dev/packages/camera

tflite: run model TensorFlow Lite
https://pub.dev/packages/tflite

provider: state management
https://pub.dev/packages/provider

2. Configure Project
  • Android
android/app/build.gradle

android {
    ...
    aaptOptions {
        noCompress 'tflite'
        noCompress 'lite'
    }
    ...
}


minSdkVersion 21

3. Load model 
loadModel() async {
    Tflite.close();
    await Tflite.loadModel(
        model: "assets/models/yolov2_tiny.tflite",  
        //ssd_mobilenet.tflite, mobilenet_v1.tflite, posenet_mv1_checkpoints.tflite
        labels: "assets/models/yolov2_tiny.txt",    
        //ssd_mobilenet.txt, mobilenet_v1.txt
        //numThreads: 1, // defaults to 1
        //isAsset: true, // defaults: true, set to false to load resources outside assets
        //useGpuDelegate: false // defaults: false, use GPU delegate
    );
  }

4. Run model

For Realtime Camera

  //YOLOv2-Tiny
  Future<List<dynamic>?> runModelOnFrame(CameraImage image) async {
     var recognitions = await Tflite.detectObjectOnFrame(
          bytesList: image.planes.map((plane) {
            return plane.bytes;
          }).toList(),
          model: "YOLO",
          imageHeight: image.height,
          imageWidth: image.width,
          imageMean: 0,                 // defaults to 127.5
          imageStd: 255.0,              // defaults to 127.5
          threshold: 0.2,               // defaults to 0.1
          numResultsPerClass: 1,
        );   
    return recognitions;
  }

  //SSDMobileNet
  Future<List<dynamic>?> runModelOnFrame(CameraImage image) async {
     var recognitions = await Tflite.detectObjectOnFrame(
          bytesList: image.planes.map((plane) {
            return plane.bytes;
          }).toList(),
          model: "SSDMobileNet",
          imageHeight: image.height,
          imageWidth: image.width,
          imageMean: 127.5,
          imageStd: 127.5,
          threshold: 0.4,
          numResultsPerClass: 1,
        );   
    return recognitions;
  }

  //MobileNet
  Future<List<dynamic>?> runModelOnFrame(CameraImage image) async {
     var recognitions = await Tflite.runModelOnFrame(
          bytesList: image.planes.map((plane) {
            return plane.bytes;
          }).toList(),
          imageHeight: image.height,
          imageWidth: image.width,
          numResults: 5
        );   
    return recognitions;
  }

  //PoseNet
  Future<List<dynamic>?> runModelOnFrame(CameraImage image) async {
     var recognitions = await Tflite.runPoseNetOnFrame(
          bytesList: image.planes.map((plane) {
            return plane.bytes;
          }).toList(),
          imageHeight: image.height,
          imageWidth: image.width,
          numResults: 5
        );   
    return recognitions;
  }

For Image

  Future<List<dynamic>?> runModelOnImage(File image) async {
    var recognitions = await Tflite.detectObjectOnImage(
        path: image.path,
        model: "YOLO",
        threshold: 0.3,
        imageMean: 0.0,
        imageStd: 127.5,
        numResultsPerClass: 1
    );
    return recognitions;
  }

Output format:

YOLO,SSDMobileNet
  [{
    detectedClass: "dog",
    confidenceInClass: 0.989,
    rect: {
        x: 0.0,
        y: 0.0,
        w: 100.0,
        h: 100.0
    }
  },...]

MobileNet
[{
    index: 0,
    label: "WithMask",
    confidence: 0.989
  },...]

PoseNet
[{
    score: 0.5,
    keypoints: {
        0: {
            x: 0.2,
            y: 0.12,
            part: nose,
            score: 0.803
        },
        1: {
            x: 0.2,
            y: 0.1,
            part: leftEye,
            score: 0.8666
        },
        ...
    }
  },...]

5. Issue 
* IOS
Downgrading TensorFlowLiteC to 2.2.0

Downgrade your TensorFlowLiteC in /ios/Podfile.lock to 2.2.0
run pod install in your /ios folder