cordova-plugin-tensorflow

Integrate the TensorFlow inference library into your PhoneGap/Cordova application!

var tf = new TensorFlow('inception-v1');
var imgData = "/9j/4AAQSkZJRgABAQEAYABgAAD//gBGRm ...";

tf.classify(imgData).then(function(results) {
    results.forEach(function(result) {
        console.log(result.title + " " + result.confidence);
    });
});

/* Output:
military uniform 0.647296
suit 0.0477196
academic gown 0.0232411
*/

Installation

Cordova

cordova plugin add https://github.com/heigeo/cordova-plugin-tensorflow

PhoneGap Build

<!-- config.xml -->
<plugin spec="https://github.com/heigeo/cordova-plugin-tensorflow.git" />

Supported Platforms

Android
iOS

API

The plugin provides a TensorFlow class that can be used to initialize graphs and run the inference algorithm.

Initialization

// Use the Inception model (will be downloaded on first use)
var tf = new TensorFlow('inception-v1');

// Use a custom retrained model
var tf = new TensorFlow('custom-model', {
    'label': 'My Custom Model',
    'model_path': "https://example.com/graphs/custom-model-2017.zip#rounded_graph.pb",
    'label_path': "https://example.com/graphs/custom-model-2017.zip#retrained_labels.txt",
    'input_size': 299,
    'image_mean': 128,
    'image_std': 128,
    'input_name': 'Mul',
    'output_name': 'final_result'
})

To use a custom model, follow the steps to retrain the model and optimize it for mobile use. Put the .pb and .txt files in a HTTP-accessible zip file, which will be downloaded via the FileTransfer plugin. If you use the generic Inception model it will be downloaded from the TensorFlow website on first use.

Methods

Each method returns a Promise (if available) and also accepts a callback and errorCallback.

classify(image[, callback, errorCallback])

Classifies an image with TensorFlow's inference algorithm and the registered model. Will automatically download and initialize the model if necessary, but it is recommended to call load() explicitly for the best user experience.

Note that the image must be provided as base64 encoded JPEG or PNG data. Support for file paths may be added in a future release.

var tf = new TensorFlow(...);
var imgData = "/9j/4AAQSkZJRgABAQEAYABgAAD//gBGRm ...";
tf.classify(imgData).then(function(results) {
    results.forEach(function(result) {
        console.log(result.title + " " + result.confidence);
    });
});

load()

Downloads the referenced model files and loads the graph into TensorFlow.

var tf = new TensorFlow(...);
tf.load().then(function() {
    console.log("Model loaded");
});

Downloading the model files can take some time. If you would like to provide a progress indicator, you can do that with an onprogress event:

var tf = new TensorFlow(...);
tf.onprogress = function(evt) {
    if (evt['status'] == 'downloading')
        console.log("Downloading model files...");
        console.log(evt.label);
        if (evt.detail) {
            // evt.detail is from the FileTransfer API
            var $elem = $('progress');
            $elem.attr('max', evt.detail.total);
            $elem.attr('value', evt.detail.loaded);
        }
    } else if (evt['status'] == 'unzipping') {
        console.log("Extracting contents...");
    } else if (evt['status'] == 'initializing') {
        console.log("Initializing TensorFlow");
    }
};
tf.load().then(...);

checkCached()

Checks whether the requisite model files have already been downloaded. This is useful if you want to provide an interface for downloading and managing TensorFlow graphs that is separate from the classification interface.

var tf = new TensorFlow(...);
tf.checkCached().then(function(isCached) {
    if (isCached) {
        $('button#download').hide();
    }
});

References

This plugin is made possible by the following libraries and tutorials:

Source	Files
TensorFlow Android Inference Interface	libtensorflow_inference.so, libandroid_tensorflow_inference_java.jar
TensorFlow Android Demo	Classifer.java, TensorFlowImageClassifier.java (modified)
TensorflowPod	Referenced via podspec
TensorFlow iOS Examples	ios_image_load.mm (modified), tensorflow_utils.mm (+ RunModelViewController.mm)