/OnnxRuntime-UnrealEngine

Apply a Style Transfer Neural Network in real time with Unreal Engine 5 leveraging ONNX Runtime.

Primary LanguageC++MIT LicenseMIT

image

This code sample is to show you how to use the new Neural Network Inference (NNI) Plugin in Unreal Engine 5 which implements ONNX Runtime to allow you to add Machine Learning (ML) Models in your projects. ONNX Runtime is a library to optimize and accelerate machine learning inferencing.

image

We are using open source models from the ONNX model zoo to apply a style transform to the scene during game play. Check out the before and after pictures below to see how one of the models is able to stylize the scene.

Before: image

After: image

Statement of Purpose

This repository aims to grow the understanding of using ML models through the NNI Plugin in Unreal Engine 5 by providing an example of implementation and references to support the Microsoft Build conference in 2022. It is not intended to be a released product. Therefore, this repository is not for discussing ONNX Runtime, the NNI plugin or requesting new features.

Prerequisites

How to run this project

Options 1: Run with Epic Games Launcher

  • Open the Epic Game Launcher and launch Unreal 5

image

  • Browse to the location that you cloned the project.
  • Select the FPStyleTransfer.uproject file and click open. This will load the same project.

image

NOTE: If you receieve the below message click Fix Now and it will recompile the project.

image

  • Once the project is loaded click play to see the project work!

image

image

Option 2: Open and run in Visual Studio.

  • Browse to the location that you cloned the project.
  • Open the FPStyleTransfer.sln file.

image

  • Hit F5 to run the project.
  • Click play to see the project work!

image

image

How to select a different model

There are 5 differnet models included from the ONNX Model Zoo. You can find them under the Content > Models folder in the UE Content Browser. To add your own models, drag and drop them into the Models folder in the Content Browser.

image

To select one of the other preloaded models or one that you added yourself.

  • Select the StyleTransferConfig blueprint from the Outliner
  • In the Details under the Default section find the Neural Network property and select the dropdown. You will see a list of models available.

image

  • Select the model you would like to apply from the dropdown, and then click play to see the new style applied.

image

Resources

Grab open source models to add to your project:

Connect with ONNX Runtime (ORT) on Social!