/zed-UE5

ZED plugin and examples for Unreal Engine 5 (Standard Engine)

Primary LanguageC++MIT LicenseMIT

ZED UE5.3 Plugin

This plugin requires using UE5.3.

This repository contains Unreal examples projects that demonstrate how to create advanced Interactive Applications using the ZED SDK and Unreal Engine 5. ZEDSamples is a single project with a variety of small samples, each meant to highlight a single feature of the ZED UE5 plugin.

Overview

Body Tracking Camera Tracking Point Cloud
ue5_bodytracking ue5_cameratracking ue5_pointcloud
Object Detection Background substraction Spatial mapping
ue5_od ue5_backgroundsub ue5_spatialmapping
Object Placement Dynamic Crosshair Hit Test
ue5_objectplacement ue5_crosshair ue5_hittest

Getting started

This version of the plugin is compatible with Unreal Engine 5.3, and requires the ZED SDK 4.1 and a Windows setup, because for now we are only compatible with the DirectX API.

Installation

  • Download or clone this repository.
  • Right click on the ZEDSamples.uproject, select switch Unreal Engine version and choose our ue5 you previously installed in the list.
  • Build the solution by pressing Ctrl + F5. Once the build is done, it will start the project in the UE5 Editor.

You can now start the project with a double click on ZEDSamples.uproject.

Troubleshooting and known issues:

  • GPU Crashed or D3D Device Removed. when playing a level: This is a DirectX 12 issue. Please change the RHI of your project to DirectX 11.
  • Error when trying to package to Shipping build: There is currently a bug preventing from building to "Shipping" build. Please use a "Development" build instead.

Documentation

A more detailed documentation is available on our website, see here : https://www.stereolabs.com/docs/ue5/

Samples

  • Body Tracking - Use 3D skeleton data to animate one or more avatars based on real time body tracking. Showcases the ZED Body Tracking feature, allowing you to track and "avatarize" people using a highly optimized AI model.

  • Camera Tracking - Use the ZED internal trackers to navigate a virtual scene by moving your ZED Camera. Explore the camera repositioning, translation multipliers and all features you can have from the Cine Camera Component.

  • Point Cloud - Generate a realtime colored point cloud of the ZED camera view, using our ultra accurate neural depth detection mode, and Unreal Engine's next-generation VFX system, Niagara.

  • Object Detection - Highlight objects detected by our Object Detection API with either a 2D frame or a 3D box.

  • Background Subtraction - Remove the background of a ZED videa stream by using either depth occlusion (put a virtual object closer to the camera than the real background, causing a view obstruction) or depth clamping (make the plugin not render any pixel further than a threshold distance).

  • Spatial Mapping - Map your environment into a 3D mesh, then shoot balls at real objects that realistically bounce off using Unreal's collision detection and physics. Optionally, save the mesh, mesh texture and .area spatial memory file for later. Use to pre-scan an environment for when you need persistent geometry, such as when collisions with the real-world need to occur even when you can't see them. See our spatial mapping documentation to learn how it works.

  • Object Placement - Click on a flat, upwards-facing surface to spawn a box above it. Demonstrates how to use the ZED's Plane Detection feature to place objects without pre-scanning. See the level Blueprint and the GetFloorPlaneAtScreenPosition function to see how it works. Press F to visualize the floor plane.

  • Dynamic Crosshair - Puts a 3D crosshair as your cursor that moves itself closer or further to be just in front of the real-world object it overlaps. Explore to see how to use the GetDepthAtScreenPosition function, and experiment with the other GetDepth and GetNormal functions included in the plugin.

  • Hit Test - Left click to shoot virtual balls that will stop when they collide with a real-world object. Demonstrates that you don't need to pre-scan an environment for such interactions thanks to the ZED's real-time depth map. See the BP_HitTestBall blueprint and the HitTestReal function to learn how it works.

Support

You will find guidance and assistance :

Bugs and fixes

You found a bug / a flaw in our plugin ? Please check that it is not already reported, and open an issue if necessary. You can also reach out to us on the community forums for any question or feedback !

By the way, we also have a special place in our hearts for PR senders.