mediapipe-iris-ios

Realtime iris tracking demo for iOS

Preview

What is MediaPipe?

MediaPipe offers cross-platform, customizable ML solutions for live and streaming media.

End-to-End acceleration: Built-in fast ML inference and processing accelerated even on common hardware
Build once, deploy anywhere: Unified solution works across Android, iOS, desktop/cloud, web and IoT
Ready-to-use solutions: Cutting-edge ML solutions demonstrating full power of the framework
Free and open source: Framework and solutions both under Apache 2.0, fully extensible and customizable

Demo

What about it?

This project is a mere copy of a demo provided by MediaPipe team. Only packaged to a dynamic framework ready to deploy on your device at one go, sparing you ~~😇painstaking😇~~ python setup.

Important: Use this project for evaluation purposes only. It is strongly recommended to build your own project/framework. MediaPipe provides streamlined build process using bazel script. This also fits well for the CI environment. You can find tutorials here.

Noise filtering

MediaPipe provides a rich set of helper classes regarding implementation of ML models. It includes math library like eigen, image processing library like opencv and many more. Most of them are abstracted for specific use cases, packed as calculators, each calculators working as a node, allowing you to create desired ML pipeline just connecting those nodes. Some of those calculators are also fine-tuned both algorithm-wise and parameter-wise, let alone the ML models being fine-tuned meaning noble paging strategy for GPU runtime and such. You don't get this kind of bargain often.

Here is one example of calculators. See the difference between the two eyes? One being stabilized and the other being fluctuating/jittering? That is because I applied a smoothing calculator to the left side of landmarks only.

It is mostly a random digital noise which makes it safe to assume the noise is somewhat Gaussian, so I chose the one-euro-filter. See how the calculator handles jittering? Try the demo yourself. All included in the bundle calculators, say no more.

One I used for this demo is LandmarksSmoothingCalculator:

# Applies smoothing to the single set of iris landmarks.
node {
  calculator: "LandmarksSmoothingCalculator"
  input_stream: "NORM_LANDMARKS:left_iris_landmarks"
  input_stream: "IMAGE_SIZE:input_image_size"
  output_stream: "NORM_FILTERED_LANDMARKS:smoothed_left_iris_landmarks"
  options: {
    [mediapipe.LandmarksSmoothingCalculatorOptions.ext] {
      one_euro_filter {
        min_cutoff: 0.01
        beta: 10.0
        derivate_cutoff: 1.0
      }
    }
  }
}

Those calculators are well documented/commented too, such as:

...
// For the details of the filter implementation and the procedure of its
// configuration please check http://cristal.univ-lille.fr/~casiez/1euro/
message OneEuroFilter {
    // Frequency of incomming frames defined in frames per seconds. Used only if
    // can't be calculated from provided events (e.g. on the very first frame).
    optional float frequency = 1 [default = 30.0];
...

Pose estimation

TODO

Build MediaPipe project as a framework

Create a BUILD file:

load("@build_bazel_rules_apple//apple:ios.bzl", "ios_framework")

ios_framework(
    name = "MPPIrisTracking",
    hdrs = [
        "MPPIrisTracker.h",
    ],
    infoplists = ["Info.plist"],
    bundle_id = "com.studio61315.MPPIrisTraking",
    families = ["iphone", "ipad"],
    minimum_os_version = "12.0",
    deps = [
        ":MPPIrisTrackingLibrary",
        "@ios_opencv//:OpencvFramework",
    ],
)

objc_library(
    name = "MPPIrisTrackingLibrary",
    srcs = [
        "MPPIrisTracker.mm",
    ],
    hdrs = [
        "MPPIrisTracker.h",
    ],
    copts = ["-std=c++17"],
    data = [
        "//mediapipe/graphs/iris_tracking:iris_tracking_gpu.binarypb",
        "//mediapipe/modules/face_detection:face_detection_front.tflite",
        "//mediapipe/modules/face_landmark:face_landmark.tflite",
        "//mediapipe/modules/iris_landmark:iris_landmark.tflite",
    ],
    sdk_frameworks = [
        "AVFoundation",
        "CoreGraphics",
        "CoreMedia",
        "UIKit"
    ],
    deps = [
        "//mediapipe/objc:mediapipe_framework_ios",
        "//mediapipe/objc:mediapipe_input_sources_ios",
        "//mediapipe/objc:mediapipe_layer_renderer",
    ] + select({
        "//mediapipe:ios_i386": [],
        "//mediapipe:ios_x86_64": [],
        "//conditions:default": [
            "//mediapipe/graphs/iris_tracking:iris_tracking_gpu_deps",
            "//mediapipe/framework/formats:landmark_cc_proto",
        ],
    }),
)

Run bazel command:

bazel build -c opt --config=ios_fat mediapipe/examples/ios/iristrackinggpuframework:MPPIrisTracking --verbose_failures

License

mediapipe-iris-ios is available under the MIT license. See the LICENSE file for more info.

Epoch2022-iOS/mediapipe-prebuilt