/MediaPipeUnityPlugin

Unity plugin to run MediaPipe graphs

Primary LanguageC#MIT LicenseMIT

MediaPipe Unity Plugin

This is a Unity (2020.3.30f1) Native Plugin to use MediaPipe (0.8.9).

The goal of this project is to port the MediaPipe API (C++) one by one to C# so that it can be called from Unity.
This approach may sacrifice performance when you need to call multiple APIs in a loop, but it gives you the flexibility to use MediaPipe instead.

With this plugin, you can

  • Write MediaPipe code in C#.
  • Run MediaPipe's official solution on Unity.
  • Run your custom Calculator and CalculatorGraph on Unity.
    • ⚠️ Depending on the type of input/output, you may need to write C++ code.

😸 Hello World!

Here is a Hello World! example.
Compare it with the official code!

using Mediapipe;
using UnityEngine;

public sealed class HelloWorld : MonoBehaviour
{
    private const string _ConfigText = @"
input_stream: ""in""
output_stream: ""out""
node {
  calculator: ""PassThroughCalculator""
  input_stream: ""in""
  output_stream: ""out1""
}
node {
  calculator: ""PassThroughCalculator""
  input_stream: ""out1""
  output_stream: ""out""
}
";

    private void Start()
    {
        var graph = new CalculatorGraph(_ConfigText);
        var poller = graph.AddOutputStreamPoller<string>("out").Value();
        graph.StartRun().AssertOk();

        for (var i = 0; i < 10; i++)
        {
            graph.AddPacketToInputStream("in", new StringPacket("Hello World!", new Timestamp(i))).AssertOk();
        }

        graph.CloseInputStream("in");
        var packet = new StringPacket();

        while (poller.Next(packet))
        {
            Debug.Log(packet.Get());
        }
        graph.WaitUntilDone().AssertOk();
    }
}

🛠️ Installation

This repository does not contain required libraries (e.g. libmediapipe_c.so, Google.Protobuf.dll, etc), so you need to build them first.
For step-by-step guide, please refer to the Installation Guide in Wiki.

⚠️ libraries that can be built differ depending on your environment.

Supported Platforms

⚠️ GPU mode is not supported on macOS and Windows.

Editor Linux (x86_64) macOS (x86_64) macOS (ARM64) Windows (x86_64) Android iOS WebGL
Linux (AMD64) 1 ✔️ ✔️ ✔️
Intel Mac ✔️ ✔️ ✔️ ✔️
M1 Mac 2 ✔️ ✔️ ✔️ ✔️
Windows 10 (AMD64) 34 ✔️ ✔️ ✔️ ✔️

🍽️ Try sample app

Example Solutions

Here is a list of solutions that you can try in the sample app.

🔔 The graphs you can run are not limited to the ones in this list.

Android iOS Linux (GPU) Linux (CPU) macOS (CPU) Windows (CPU) WebGL
Face Detection ✔️ ✔️ ✔️ ✔️ ✔️ ✔️
Face Mesh ✔️ ✔️ ✔️ ✔️ ✔️ ✔️
Iris ✔️ ✔️ ✔️ ✔️ ✔️ ✔️
Hands ✔️ ✔️ ✔️ ✔️ ✔️ ✔️
Pose ✔️ ✔️ ✔️ ✔️ ✔️ ✔️
Holistic ✔️ ✔️ ✔️ ✔️ ✔️ ✔️
Selfie Segmentation
Hair Segmentation ✔️ ✔️ ✔️ ✔️ ✔️ ✔️
Object Detection ✔️ ✔️ ✔️ ✔️ ✔️ ✔️
Box Tracking ✔️ ✔️ ✔️ ✔️ ✔️ ✔️
Instant Motion Tracking ✔️ ✔️ ✔️ ✔️ ✔️ ✔️
Objectron ✔️ ✔️ ✔️ ✔️ ✔️ ✔️
KNIFT

UnityEditor

Select Mediapipe/Samples/Scenes/Start Scene and play.

Desktop

If you've built native libraries for CPU (i.e. --desktop cpu), select CPU for inference mode from the Inspector Window. preferable-inference-mode

Android, iOS

Make sure that you select GPU for inference mode before building the app, because CPU inference mode is not supported currently.

📖 Wiki

🚧 https://github.com/homuler/MediaPipeUnityPlugin/wiki

📜 LICENSE

MIT

Note that some files are distributed under other licenses.

Footnotes

  1. Tested on Arch Linux. ↩

  2. Experimental, because MediaPipe does not support M1 Mac. ↩

  3. Windows 11 will be also OK. ↩

  4. Running MediaPipe on Windows is experimental. ↩