flutter_tflite_audio

This plugin allows you to use tflite to make audio/speech classifications. Can now support ios and android.

If there are any problems with the plugin, please do not hesistate to create an issue or request features on github.

Limitations of this plugin and roadmap

Can only use decoded wave for the tensor input. Will add an ability to use mfcc in the future.
Can only run the model once after recording. Will add a feature to run the model multiple times or indefintely the user's specification.
Fixed tensor input and output shape. Perhaps will add a feature where this can be shaped.

How to add tflite_audio as a dependency:

Add tflite_audio as a [dependency in your pubspec.yaml file]

How to add tflite model and label to flutter:

Place your custom tflite model and labels into the asset folder.
In pubsec.yaml, link your tflite model and label under 'assets'. For example:

  assets:
    - assets/conv_actions_frozen.tflite
    - assets/conv_actions_labels.txt

How to use this plugin

Please look at the example on how to implement these futures.

Import the plugin. For example:

import 'package:tflite_audio/tflite_audio.dart';

To load your model, bascially call the the future loadModel() and assign the appropriate values to the arguments like below:

//Loads your model
 Future loadModel({model, label, numThreads, isAsset}) async {
    return await TfliteAudio.loadModel(model, label, numThreads, isAsset);
  }

loadModel(
        model: "assets/conv_actions_frozen.tflite",
        label: "assets/conv_actions_labels.txt",
        numThreads: 1,
        isAsset: true);

To get the results, call the future startAudioRecognition and assign the appropriate values to the arguments.

  Future<Map<dynamic, dynamic>> startAudioRecognition(
      {int sampleRate, int recordingLength, int bufferSize}) async {
    return await TfliteAudio.startAudioRecognition(
        sampleRate, recordingLength, bufferSize);
  }

//To get the results from invoking the model
  Map<dynamic, dynamic> result = await startAudioRecognition(
            sampleRate: 16000, recordingLength: 16000, bufferSize: 1280);

//Call values from map
  var recognitionResult = result['recognitionResult'] 
  var inferenceTIme = result['inferenceTime']

For a rough guide on the parameters

numThreads - higher threads will reduce inferenceTime. However, cpu usage will be higher.

sampleRate - determines the number of samples per second

recordingLength - determines the max length of the recording buffer. If the value is not below or equal to your tensor input, it will crash.

bufferSize - Make sure this value is equal or below your recording length. A very high value may not allow the recording enough time to capture your voice. A lower value will give more time, but it'll be more cpu intensive

Android Installation & Permissions

Add the permissions below to your AndroidManifest. This could be found in /android/app/src folder. For example:

<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />

Edit the following below to your build.gradle. This could be found in /app/src/For example:

aaptOptions {
        noCompress 'tflite'

iOS Installation & Permissions

Add the following key to Info.plist for iOS. This ould be found in /ios/Runner

<key>NSMicrophoneUsageDescription</key>
<string>Record audio for playback</string>

Change the deployment target to at least 12.0. This could be done by:

a. Open your project workspace on xcode

b. Select root runner on the left panel

c. Under the info tab, change the iOS deployment target to 12.0
Open your podfile in your iOS folder and change platform ios to 12. Also make sure that use_frameworks! is under runner. For example

platform :ios, '12.0'

target 'Runner' do
  use_frameworks! #Make sure you have this line
  use_modular_headers!

  flutter_install_all_ios_pods File.dirname(File.realpath(__FILE__))
end

References

https://github.com/tensorflow/examples/tree/master/lite/examples/speech_commands

Alii-isk/flutter_tflite_audio