Dracarys Interactive supplies Unity-based tools and templates for creating quality interactive 2D, 3D, and XR applications.
- Introduction
- Using AI Studio
- AI Studio Dependencies
- Running the Simple Scenes
- Running the City Scenes
- Running Avatar Wars
- Support
AI Studio provides a flexible open-source framework for integrating Unity games and applications with AI systems that enable real-time, speech-driven dialogue and actions. The core framework may be integrated with your choice of speech and text recognition and generation models and APIs.
This framework is a byproduct of a case study exploring a step-by-step integration of OpenAI's GPT-3.5 model into Unity 3D for the purpose of enhancing non-playing characters (NPCs) in video games and interactive applications. By leveraging GPT-3.5's natural language processing capabilities, we aimed to create NPCs capable of dynamic real-time interactions with players and each other! Our methodology involved integrating GPT-3.5, implementing speech-driven dialogues, and incorporating humanoid avatars with lip-syncing and emotive animations.
Key features of AI Studio include:
- Core action-driven architecture implementing multiple character dialogues which feeds asynchronous events into a central queue for processing by the main Unity thread
- Basic Scriptable Object framework for representing characters and character dialogues and providing AI model prompt engineering inputs
- Abstracted interfaces to key implementation boundaries including speech recognition and synthesis, natural language dialogue generation, and dialogue character animation and movement
- Feature extensions including streaming and non-streaming audio, continuous and discrete player voice recognition, conditional dialogue triggering, player synthetic voice option, programmatic chat injection, and basic non-player character movement
- Example scenes demonstrating integration with Microsoft Cognitive Services Speech SDK, OpenAI's chat completion models (e.g. GPT-3.5), Unity Multipurpose Avatars (UMA), SALSA LipSync Suite, and Ready Player Me avatars
- YouTube videos documenting installation and example scenes
AI Studio has several dependencies that require licensing and importing before all the sample scenes can be compiled and run. The target platform for running sample scenes is a Windows 10 or later PC running Unity 2020.3 or later. There are two sets of sample scenes included in AI Studio. “Simple” scenes demonstrate the core architecture and “City” scenes with additional dependencies add a game world and humanoid avatars. One “City” scene, Avatar Wars, adds an additional dependency on Ready Player Me.
The AI Studio package itself includes no third-party content except for a couple of Unity Starter Asset scripts. However, to run the sample scenes in AI Studio, several third-party licenses and packages must be acquired and installed by the user. These packages are not included in the AI Studio package and until installed the sample scenes will contain objects with missing component scripts. Once the required packages are installed, these missing component script errors will be resolved. Instructions for acquiring and installing these packages are included in this README and are also documented in a set of YouTube videos. The complete set of dependencies is shown below.
To run the “Simple” sample scenes, the following dependencies must be satisfied:
- An OpenAI API key.
- Integration for the OpenAI API in Unity. This is an open-source Unity package hosted in GitHub governed by the MIT license.
- A subscription key for the Microsoft Azure Speech Service.
- The Microsoft Cognitive Services Speech SDK for Unity package.
To run the “City” sample scenes except for “Avatar Wars”, these additional packages must be installed:
- The Unity Cinemachine package from the Unity Registry.
- The UMA 2 Unity Multipurpose Avatar package from the Unity Asset Store.
- The CITY package from the Unity Asset Store.
- The SALSA Lip Sync Suite from Unity Asset Store.
- The SALSA Lip Sync Suite OneClick UMA DCS package from the Crazy Minnow Studio website.
- A package of several Mixamo animations.
To run the “City” sample scene “Avatar Wars”, these additional dependencies must be satisfied:
- The Ready Player Me Unity SDK package.
The next section describes in more detail how to satisfy these dependencies and run the AI Studio sample scenes.
To run the Simple scenes which demonstrate the core architecture, the following licenses must be obtained, and accompanying Unity packages must be installed.
- Download and install the Dracarys Interactive AI Studio package from the Unity Asset Store or via GitHub or from here.
- You may get missing Prefab errors due to the dependencies .../Prefabs/Environment.prefab has on the CITY package, but this can be ignored when running the Simple scenes.
- An OpenAI account and API key are required to run all sample scenes. Once you have obtained the key, you must create a folder .openai under your home folder with a file auth.json containing the following:
{ private_api_key : "YOUR_KEY_HERE" }
- Alternatively, you can modify the OpenAI Unity API authentication type from “Local File” to “String” and enter your OpenAI API key directly via the Unity Inspector.
- Add and import the OpenAI Unity API package from GitHub.
- Add the Scripting Define Symbol USE_COM_OPENAI_API_UNITY to the Player Project Settings under Script Compilation.
- A subscription key for the Microsoft Azure Speech Service.
- Once you have obtained the key, you must set the environment variables COGNITIVE_SERVICE_KEY and COGNITIVE_SERVICE_REGION.
- Alternatively, you can enter the key and region fields directly via the Inspector.
- Download and install the Speech SDK for Unity Package. You may also download this package from here.
- Add the Scripting Define Symbol USE_MICROSOFT_COGNITIVESERVICES_SPEECH to the Player Project Settings under Script Compilation.
There are three Simple scenes that you can run under Skylands Research Institure/AI Studio/Scenes:
- Simple 2 NPC - This scene demonstrates dialogue between two NPCs.
- Simple 3P Player and 1 NPC - One third-person player and one non-player.
- Simple 3P Player and 1 NPC - One third-person player and two non-players.
This YouTube video demonstrates how to create a Unity project from scratch that can run the Simple example scenes.
To run the City scenes which include a city game world and avatar characters, the following licenses must be obtained, and accompanying Unity packages must be installed.
- The Cinemachine package may be imported via the Unity Package Manager through the Unity Registry.
- Add the Scripting Define Symbol USE_CINEMACHINE to the Player Project Settings under Script Compilation.
NOTE! Unity Starter Assets First and Third-person package scripts are included in the AI Studio package so the following steps to import and modify these packages are for information only. It should not be necessary to import these packages. However you will need to add the Scripting Define Symbol STARTER_ASSETS_PACKAGES_CHECKED to the Player Project Settings under Script Compilation.
- The Unity Asset Store Starter Assets - Third Person Character Controller, a free asset, must be imported and a minor C# script change must be applied to make the controller work with UMA
- Accept the import of any dependencies.
- Changes to StarterAssets\Scripts\ThirdPersonController.cs
- Expose _animator field (line 104):
[SerializeField] private Animator _animator;
- Use attached Animator if set (line 139 and 158):
if (!(_hasAnimator = _animator)) _hasAnimator = TryGetComponent(out _animator);
- Make method OnFootstep public (line 372)
- Make method OnLand public (line 384)
- Expose _animator field (line 104):
- The Unity Asset Store Starter Assets - First Person Character Controller, a free asset, must be imported.
- The Unity Starter Assets depend on the new input system so you will need to make to set the Player settings active input handling to either the new input system or both the new and the old.
- The Input System must be installed from the Unity Registry
- The Unity Asset Store UMA 2 - Unity Multipurpose Avatar, a free asset, must be imported.
- The Unity Asset Store CITY package, a free asset, must be imported.
- The Unity Asset Store SALSA LipSync Suite, a paid asset, must be imported.
- NOTE! You will also need to download and install the OneClick UMA DCS add-on package from here.
- SALSA will automatically add the Scripting Define Symbol CMS_SALSA to the Player Project Settings under Script Compilation.
- Import a set of Mixamo animations from here.
This YouTube video demonstrates how to extend the Unity project configured to run the Simple scenes to run the City scenes with the exception of Avatar Wars which will be covered in the next section.
The scene City Scene Avatar Wars 2 NPC requires the installation of Ready Player Me.
- Follow these instructions to setup RPM and install the software into Unity. Make sure to also import all the loader samples.
- Import an RPM avatar package from here.
This YouTube video demonstrates how to extend the Unity project configured to run the City scenes to run Avatar Wars.
If you have questions or issues with AI Studio please visit the Dracarys Interactive website and use the Contact page to get in touch with us.