MediaPipe Unity Plugin

This is a Unity (>= 2021.3) Native Plugin to use MediaPipe (0.10.14).

The goal of this project is to port the MediaPipe API (C++) one by one to C# so that it can be called from Unity.
This approach may sacrifice performance when you need to call multiple APIs in a loop, but it gives you the flexibility to use MediaPipe instead.

With this plugin, you can

Write MediaPipe code in C#.
Run MediaPipe's official solution on Unity.
Run your custom Calculator and CalculatorGraph on Unity.
- ⚠️ Depending on the type of input/output, you may need to write C++ code.

😸 Hello World!

Here is a Hello World! example.
Compare it with the official code!

using Mediapipe;
using UnityEngine;

public sealed class HelloWorld : MonoBehaviour
{
    private const string _ConfigText = @"
input_stream: ""in""
output_stream: ""out""
node {
  calculator: ""PassThroughCalculator""
  input_stream: ""in""
  output_stream: ""out1""
}
node {
  calculator: ""PassThroughCalculator""
  input_stream: ""out1""
  output_stream: ""out""
}
";

    private void Start()
    {
        using var graph = new CalculatorGraph(_ConfigText);
        using var poller = graph.AddOutputStreamPoller<string>("out");
        graph.StartRun();

        for (var i = 0; i < 10; i++)
        {
            graph.AddPacketToInputStream("in", Packet.CreateStringAt("Hello World!", i));
        }

        graph.CloseInputStream("in");
        var packet = new Packet<string>();

        while (poller.Next(packet))
        {
            Debug.Log(packet.Get());
        }
        graph.WaitUntilDone();
    }
}

For more detailed usage, see the API Overview page or the tutorial on the Getting Started page.

🛠️ Installation

This repository does not contain required libraries (e.g. libmediapipe_c.so, Google.Protobuf.dll, etc).
You can download them from the release page instead.

file	contents
`MediaPipeUnityPlugin-all.zip`	All the source code with required libraries. If you need to run sample scenes on your mobile devices, prefer this.
`com.github.homuler.mediapipe-*.tgz`	A tarball package
`MediaPipeUnityPlugin.*.unitypackage`	A `.unitypackage` file

If you want to customize the package or minify the package size, you need to build them by yourself.
For a step-by-step guide, please refer to the Installation Guide on Wiki.
You can also make use of the Package Workflow on Github Actions after forking this repository.

⚠️ libraries that can be built differ depending on your environment.

Supported Platforms

⚠️ GPU mode is not supported on macOS and Windows.

	Editor	Linux (x86_64)	macOS (x86_64)	macOS (ARM64)	Windows (x86_64)	Android	iOS
Linux (AMD64) ¹	✔️	✔️				✔️
Intel Mac	✔️		✔️			✔️	✔️
M1 Mac	✔️			✔️		✔️	✔️
Windows 10/11 (AMD64) ²	✔️				✔️	✔️

Supported Solutions

This plugin implements the following MediaPipe Tasks C# APIs.

cf. The official available solutions

Solution	Android	iOS	Linux	macOS	Windows
LLM Inference API
Object detection	✔️	✔️	✔️	✔️	✔️
Image classification
Image segmentation	✔️	✔️	✔️	✔️	✔️
Interactive segmentation
Hand landmark detection	✔️	✔️	✔️	✔️	✔️
Gesture recognition
Image embedding
Face detection	✔️	✔️	✔️	✔️	✔️
Face landmark detection	✔️	✔️	✔️	✔️	✔️
Face stylization
Pose landmark detection	✔️	✔️	✔️	✔️	✔️
Image generation
Text classification
Text embedding
Language detector
Audio classification	✔️	✔️	✔️	✔️	✔️

Legacy Solutions

You can also use MediaPipe Framework, which allows you to run Legacy Solutions. However, please note that support for these solutions has ended.

🍽️ Try the sample app

Example Solutions

Some solutions (including Legacy solutions) can be tested using the sample app. Please check Assets/MediaPipeUnity/Samples/Scenes to see which solutions have samples.

UnityEditor

Select any scenes under Assets/MediaPipeUnity/Samples/Scenes and play.

Desktop, Android, iOS

Select proper Inference Mode and Asset Loader Type from the Inspector Window.

Preferable Inference Mode

If the native libraries are built for CPU (i.e. --desktop cpu), select CPU for inference mode.
For the libraries distributed on the release page, only the CPU is available for use on Windows and macOS.

Asset Loader Type

The default Asset Loader Type is set to Local, which only works on UnityEditor.
To run it on your devices, switch it to StreamingAssets and copy the required resources under StreamingAssets (if you're using MediaPipeUnityPlugin-all.zip, the StreamingAssets directory already contains them).

See the tutorial for more details.

📖 Wiki

https://github.com/homuler/MediaPipeUnityPlugin/wiki

📜 LICENSE

MIT

Note that some files are distributed under other licenses.

MediaPipe (Apache Licence 2.0)
emscripten (MIT)
- third_party/mediapipe_emscripten_patch.diff contains code copied from emscripten
FontAwesome (LICENSE)
- Sample scenes use Font Awesome fonts