This repo is intended to provide more advanced demos for AR Foundation outside of the Samples Repo.
For questions and issues related to AR Foundation please post on the AR Foundation Sample issues and NOT in this repo. You can also post on the AR Foundation Forums
AR Foundation demo projects.
Demo projects that use AR Foundation 3.0 and demonstrate more advanced functionality around certain features
This set of demos relies on five Unity packages:
- ARSubsystems (documentation)
- ARCore XR Plugin (documentation)
- ARKit XR Plugin (documentation)
- ARKit Face Tracking (documentation)
- ARFoundation (documentation)
ARSubsystems defines an interface, and the platform-specific implementations are in the ARCore and ARKit packages. ARFoundation turns the AR data provided by ARSubsystems into Unity GameObjects and MonoBehavours.
The master branch is compatible with Unity 2019.3
| Image Tracking | Onboarding UX | Mesh Placement | Shaders |
|---|
Image Tracking — Also available on the asset store here
A sample app showing off how to use Image Tracking to track multiple unique images and spawn unique prefabs for each image.
The script ImageTrackingObjectManager.cs. handles storing prefabs and updating them based on found images. It links into the ARTrackedImageManager.trackedImagesChanged callback to spawn prefabs for each tracked image, update their position, show a visual on the prefab depending on it's tracked state and destroy it if removed.
The project contains two unique images one.png two.png which can be printed out or displayed on digital devices. The images are 2048x2048 pixels with a real world size of 0.2159 x 0.2159 meters.
The Prefabs for each number are prefab variants derived from OnePrefab.prefab. They use a small quad that uses the MobileARShadow.shader in order to accurately show a shadow of the 3D number.
The script DistanceManager.cs checks the distances between the tracked images and displays an additional 3D model between them when they reach a certain proximity.
the script NumberManager.cs handles setting up a contraint (in this case used to billboard the model) on the 3D number objects and provides a function to enable and disabling the rendering of the 3D model.
If you import the image tracking package or download it from the asset store without the Onboarding UX there will be a Missing Prefab in your scene. This prefab is a configured ScreenSpaceUI prefab from the Onboarding UX. It is configured with the UI for finding an image with the goal of finding an image.
UX — Also available on the asset store here
A UI / UX framework for providing guidance to the user for a variety of different types of mobile AR apps.
The framework adopts the idea of having instructional UI shown with an instructional goal in mind. One common use of this is UI instructing the user to move their device around with the goal of the user to find a plane. Once the goal is reached the UI fades out. There is also a secondary instruction UI and an API that allows developers to add any number of additional UI and goals that will go into a queue and be processed one at a time.
A common two step UI / Goal is to instruct the user to find a plane. Once a plane is found you can instruct the user to tap in order to place an object. Once an object is placed fade out the UI.
The instructional UI consist of the following animations / videos
- Cross Platform Find a plane
- Find a Face
- Find a Body
- Find an Image
- Find an Object
- ARKit Coaching Overlay
- Tap to Place
- None
All of the instructional UI (except the ARKit coaching overlay) is an included .webm video encoded with VP8 codec in order to support transparency.
With the following goals to fade out UI
- Found a plane
- Found Multiple Planes
- Found a Face
- Found a Body
- Found an Image
- Found an Object
- Placed an Object
- None
The goals are checking the associated ARTrackableManager number of trackables count. One thing to note is this is just looking for a trackable to be added, it does not check the tracking state of said trackable.
The script UIManager.cs is used to configure the Instructional Goals, secondary instructional goals and holds references to the different trackable managers.
UIManager manages a queue of UXHandle which allows any instructional UI with any goal to be dynamically added at runtime. To do this you can store a reference to the UIManager and call AddToQueue() passing in a UXHandle object. For testing purposes to visualize every UI video I use the following setup.
m_UIManager = GetComponent<UIManager>();
m_UIManager.AddToQueue(new UXHandle(UIManager.InstructionUI.CrossPlatformFindAPlane, UIManager.InstructionGoals.PlacedAnObject));
m_UIManager.AddToQueue(new UXHandle(UIManager.InstructionUI.FindABody, UIManager.InstructionGoals.PlacedAnObject));
m_UIManager.AddToQueue(new UXHandle(UIManager.InstructionUI.FindAFace, UIManager.InstructionGoals.PlacedAnObject));
m_UIManager.AddToQueue(new UXHandle(UIManager.InstructionUI.FindAnImage, UIManager.InstructionGoals.PlacedAnObject));
m_UIManager.AddToQueue(new UXHandle(UIManager.InstructionUI.FindAnObject, UIManager.InstructionGoals.PlacedAnObject));
m_UIManager.AddToQueue(new UXHandle(UIManager.InstructionUI.ARKitCoachingOverlay, UIManager.InstructionGoals.PlacedAnObject));
There's a m_CoachingOverlayFallback used in order to enable the ARKit coaching overlay on supported devices but fall back to Cross Platform Find a Plane when it is not.
The script ARUXAnimationManager.cs holds references to all the videos, controls all the logic for fading the UI in and out, managing the video swapping and swapping the associated text with each video / UI.
The script DisableTrackedVisuals holds a reference to the ARPlaneManger and ARPointCloudManager to allow for disabling both the spawned objects from the managers and the managers themselves, preventing further plane tracking or feature point (point clouds) tracking.
When the session (device) is not tracking or has lost tracking there are a variety of different reasons why. It can be helpful to show these reasons to users so they better understand the experience or what may be hindering it.
Both ARKit and ARCore have slightly different reasons but in AR Foundation these are surfaced through the same shared API.
The ARUXReasonsManager.cs handles the visualization of the states and subscribes to the state change on the ARSession. The reasons are set and the display text and icon are changed in the SetReaons() method. Here I treat both Initializing and Relocalizing the same and for english display Initializing augmented reality.
If you want to use localization make sure to read the required addressables building documentation at the end of this section.
The Instructional UI and the Reasons have localization support through the Unity localization package. It's enabled for the the instructional UI in AR UX Animation Manager with the m_LocalizeText bool and with reasons in the AR UX Reasons Manager with the m_LocalizeText bool.
Localization currently supports the following languages
- English
- French
- German
- Italian
- Spanish
- Portuguese
- Russian
- Simplified Chinese
- Korean
- Japanese
- Hindi
- Dutch
Tamil and Telugu translations are available but due to font rendering complexities are not enabled currently
The localizations are supported through a CSV that is imported into the project and parsed into the proper localization table via StringImporter.cs.
If you would like to help out, have a suggestion for a better translation or want to add additional languges please reach out and comment on this publicly available Sheet
In the scene Localization is driven by the script LocalizationManager.cs which has a SupportedLanguages enum for each supported language. The current implementation only supports selecting and setting a language at compile time and NOT at runtime. This is because the selected language from the enum is set in the Start() method of LocalizationManager.cs.
After the language is set the localized fields are retrieved from the tables based on specific keys for each value and then referenced in the AR UX Animation Manager and AR UX Reasons Manager.
Many languages require unique fonts in order to properly render the characters for these languages the font's are swapped at runtime along with language specific settings in SwapFonts()
The Localization package uses Addressables to organize and pack the translated strings. There are some additional steps required to properly build these for your application. If you're localizing the text for the instructions or the reasons you will need to do these steps.
- Open the Addressables Groups window (Window / Asset Management / Addressables / Groups)
- In the Addressables Groups Window click on the Build Tab / New Build / Default Build Script
- You will need to do this for every platform you are building for. (Once for Android and once for iOS).
An example scene for using ARKit meshing feature with the available surface classifications to place unique objects on surfaces. This demo adds some additional functionality for use cases helpful outside of this demo such as a placement reticle and the DOTween tweening library.
Classifying the surfaces is managed by the MeshClassificationManager.cs which maintains a Dictionary of TrackableID's and a native array of ARMeshClassifications. By subscribing to the meshesChanged event on the AR Mesh Manager we maintain the dictionary of added, updated and removed meshes based on trackable ID's of the meshes generated.
there is currently an issue with the trackable ID's on the meshes found so we use the string name is order to properly extract and store the correct trackable ID of the meshes
Once we have an up to date dictionary we can query it based on a trackable ID as the key and a triangle index as the index in our native array. This returns an ARMeshClassification enum.
To update the label at the top of the demo we use a physics raycast to raycast against the megamesh generated by the ARMeshManager to get the correct triangle index and parse the current classification for a more readable string label.
to generate a mesh collider for physics raycast our megamesh must contain a mesh collider component on it
The Mesh Placement Manager script handles showing the UI for each unique surface and spawning the objects at the placement reticle position. In the Update method I am checking against specific classifications, in this case Table, Floor and Wall to enable or disable specific UI buttons. The UI buttons are configured in the scene to pass an index and instantiate the assigned prefab in the object list for each surface.
There's also some additional logic for placing floor and table objects to rotate them towards the user (Camera transform).
A way to place content on surfaces based on the center screen position of the users device. This reticle shows a visual that can snap to mesh (generated ARKit mesh) or planes. It uses an AR raycast to find the surfaces and snaps to AR Raycast Hit pose position and rotation.
There is also additional logic to scale up the reticle's local scale based on the distance away from the user (AR Camera transform).
For determining between snapping to a mesh and a plane we use a Raycast Mask.
Mesh:
m_RaycastMask = TrackableType.PlaneEstimated;
Plane:
m_RaycastMask = TrackableType.PlaneWithinPolygon;
To visualize and understand the different classified surfaces we are using a modified version of the MeshFracking script available in AR Foundaiton Samples. We've added an additional helper method to modify the alpha color of the generated meshes ToggleVisability(). This is all driven by a Toggle UI button in the scene and changes the shared material color on each material on the generated prefabs. By default they are configured to be completely transparent.
DOTween is available on the Unity Asset store here
For this demo it is used to scaling up the placed objects as they appear.
It was developed by Daniele Giardini - Demigiant and is Copyright (c) 2014. Full License for DOTween available here
A collection of Shaders built for AR and AR use cases
This effect uses the latest Depth API and is only available on LiDAR enabled iOS devices like the iPad Pro. Currently this is only supported in the built-in render pipeline.
The Fog scene uses an ARKit background shader that incorporates Unity scene fog into the shader used for rendering the ARKit device camera feed to the screen. It is heavily based on the ARKitBackground shader shipped with the ARKit package.
In order to properly use this shader the Custom Material checkbox on the ARCameraBackground component must be checked and a material with this shader assigned.
The Fog scene includes AR plane finding and placement scritps to place a virtual object in AR that uses the standard shader. The scene has fog enabled in the Lighting Settings window. It's set to an end distance of 35 and a UI slider that changes the scene fog between 1-35 is configured in the scene. Manipulating this slider will change the appearance of the density of the fog in the scene.
These shaders are configured with Universal Render Pipeline and Shader Graph. They are expected to change or potentially break if either package is updated. Some of the implementations of these shaders are based on current API's and package structures that are not guaranteed to be consistent in future packages.
To enable these shaders you must assign the Universal Render Pipeline Asset in Project Graphics settings. This Pipeline asset has been pre-configured to work well with the shadow shaders and the ARFoundationForwardRendererData asset has been configured to properly render with AR Foundation by adding the AR Background Render Feature.
This shader enables visualizing the edges of any mesh in Unity. Here we are using it to visualize the runtime mesh generated by ARKit from the AR Mesh Manager. This shader works by combining barycentric data (vertex colors) with a custom Shader graph for determining edges and painting pixels an assigned color.
The MeshVisualization scene is configured to apply the barycentric data at runtime by subscribing to the MeshChanged events in the AR Mesh Manager component. The AR Mesh Manager uses the Mesh Visualization prefab that has a Mesh Filter and Mesh Renderer with an assigned material using the WireFrame shader graph. When meshes are added or updated the barycentric data also gets updated on the mesh enabling this effect.
This sample enables and disables plane tracking by Toggling the AR Plane Manager component. You can see the optimizations with the mesh along flat surfaces when plane tracking is enabled.
Shadows are important for grounding objects in AR. It helps the user better understand the depth and position of augmented content in the real world. For enabling shadows in the Universal Render Pipeline custom .hlsl files have been written and used in shader graphs as custom nodes.
Both of these shaders are built to be applied on a single flat surface such as a plane or quad. For heirarchy setup you can see that the prefabs using these shaders have an empty root game object, the plane at the base of the object and the content positioned above that both as children of the root object.
The blurred shadows have two custom node inputs. One for contact shadows created with objects close to the plane using this shader. Another for implementing stocastic blurring to create much softer shadows giving it a blurred edge. It is recommended to use these shadows for dynamic content.
This uses a custom node to consume the lighting data in the scene and apply it to a transparent surface. The shadows of this shader are driven by the graphics and quality settings in the project. These are the level of shadows you can expect out of the box from Unity. It is recommended to use these shadows for static content.
Camera grain is a unique feature to ARKit which produces a tileable metal texture to match the visual characteristics of the current video stream. In Unity this is surfaced as a 3D grain texture through the ARCameraFrameEventArgs. For the shader sample this grain texture is then applied to a custom shader graph that also creates visual noise on the object. This effect in general is very subtle and more visible in darker areas where the grain on the camera feed is also more apparent.
Compatibility for 2020.2+ version of Unity is due to how 3D textures are handled and managed
This sample also uses a script ProbePlacement for manually placing environmental probes to further enhance the effect.