flutter_d2go

Flutter Plugin inferring using d2go, the mobile model of detectron2.

Features

Get class and boundary box by object detection (Android and iOS)
Get keypoints by keypoint estimation (Android and iOS)
Get mask data by instance segmentation (Android only)
Live inference for camera stream images (Android only)

Preview

Live inference for camera stream images
Object detection and instance segmentation
Keypoints estimation

Installation

Add flutter_d2go to your pubspec.yaml.

Put the d2go model and class file in the assets directory.

assets:
  - assets/models/d2go.pt
  - assets/models/classes.txt

Usage

1. Load model and classes

The model is in Pytorch format.
The format of classes file is here.

await FlutterD2go.loadModel(
    modelPath: 'assets/models/d2go.pt',     // required
    labelPath: 'assets/models/classes.txt', // required
);

2. Get static image predictions

List<Map<String, dynamic>> output = await FlutterD2go.getImagePrediction(
    image: image,           // required File(dart:io) image
    width: 320,             // defaults to 320
    height: 320,            // defaults to 320
    mean: [0.0, 0.0, 0.0],  // defaults to [0.0, 0.0, 0.0]
    std: [1.0, 1.0, 1.0],   // defaults to [1.0, 1.0, 1.0]
    minScore: 0.7,          // defaults to 0.5
);

3. Get stream images predictions

List<Map<String, dynamic>> output = await FlutterD2go.getStreamImagePrediction(
    imageBytesList: cameraImage.planes.map((plane) => plane.bytes).toList(),             // required List<Uint8List> image byte array
    imageBytesPerPixel: cameraImage.planes.map((plane) => plane.bytesPerPixel).toList(), // default to [1, 2, 2]
    width: cameraImage.width,               // default to 720
    height: cameraImage.height,             // default to 1280
    inputWidth: 320,                        // defaults to 320
    inputHeight: 320,                       // defaults to 320
    mean: [0.0, 0.0, 0.0],                  // defaults to [0.0, 0.0, 0.0]
    std: [1.0, 1.0, 1.0],                   // defaults to [1.0, 1.0, 1.0]
    minScore: 0.7,                          // default to 0.5
    rotation: 90,                           // default to 0
);

Predictions `output` format

rect is the scale of the original image.
mask and keypoints depend on whether the d2go model has mask and keypoints.

mask will be a Uint8List of bitmap images bytes. keypoints will be a list of 17 (x, y).

[
  {
    "rect": {
      "left": 74.65713500976562,
      "top": 76.94147491455078,
      "right": 350.64324951171875,
      "bottom": 323.0279846191406
    },
    "mask": [66, 77, 122, 0, 0, 0, 0, 0, 0, 0, 122, ...],
    "keypoints": [[117.14504, 77.277405], [122.74037, 73.53044], [105.95437, 73.53044], ...],
    "confidenceInClass": 0.985002338886261,
    "detectedClass": "bicycle"
  }, // For each instance
...
]

Issues

Please contact here if you find any bugs or features you would like to add.

tsubauaaa/flutter_d2go