Maclean-D/three-assistant-glass

Customizeable 3D conversational AI character with Vapi, three-vrm, & Looking Glass support.

three-assistant-glass

Customizable 3D conversational AI character

Features

Voice Assistant - Vapi

• Custom LLM (OpenAI, Anthropic, Groq, etc.)
• Custom Voice (Cartesia, 11labs, Rime.ai etc.)
• Knowledge Base (Markdown, PDF, Word, jpeg, etc.)
• Live Transcriptions (Deepgram, Talkscriber, Gladia)
• Emotion Detection
• Interuptions
• Background Sound, Filler, & Backchanneling
• Function Calling
• Audio Recording

Character - three-vrm

• Custom 3D model (vrm)
• Custom animations (fbx)
• Animated to voice assistant's voice

Holographic Display - Looking Glass

• View in 3d on any Looking Glass display

Experimental

• Plaintext clipboard access

Planned

• Show current time
• Add setting to change size/scale of character
• Add setting to move character backwards or forwards
• Access to Ask Limitless via Selenium upon public release
• 01 as a voice assistant option upon a stable 1.0 release

Prerequisites

• Desktop operating system (Windows, MacOS, Linux)
• Node.js
• npm (usually comes with Node.js)

How to Run

1. Open a terminal and clone this repository

   git clone https://github.com/Maclean-D/three-assistant-glass.git
   

2. Navigate to the project directory.

   cd three-assistant-glass
   

3. Install the required dependencies:

   npm install
   

4. Start the server:

   node server.mjs
   

5. http://localhost:3000/ should open automatically

6. Open Settings and save your Vapi keys

7. Pick a character model and voice assistant (Create an assistant on Vapi first if you haven't already)

8. Go back to http://localhost:3000/settings and click ▶️ to start the assistant

View on a Looking Glass Display

1. Install Looking Glass Bridge

2. Plug in your Looking Glass Display

3. (Recommended) Turn off Show Settings Icon in Settings

4. Click Enter Looking Glass

5. Double click the window on the Looking Glass Display or press f11 to enter full-screen

6. Press the ▶️ button on your looking glass display or computer to start the assistant (Configurable in Settings)

FAQ

How do I add a new character model?

1. Download a .vrm file, (VRoid Hub has lots of models or make your own in VRoid Studio)
2. From the characters tab in settings, click the + button and select the .vrm or .zip file
3. Click the model to set it as the active model

How do I change the character's card in settings?

1. Prepare a 270x480 .png file that has the same name as your vrm file
2. From the characters tab in settings, click the + button and select the .png file

Can I bundle character models and cards?

1. Zip the character's .vrm & .png files together (make sure they have the same name)
2. From the characters tab in settings, click the + button and select the .zip file

How do I change a character's name?

1. Open the characters folder
2. Rename the .vrm & .png files (make sure they have the same name)

How do I delete characters?

1. Open the characters folder
2. Delete the desired .vrm & .png files

How do I add custom animations?

1. Open the animations folder
2. Drag and drop your .fbx files into the folder

I turned off the Settings Icon, how do I get it back?

1. Go back to http://localhost:3000/settings
2. Turn on Show Settings Icon

Troubleshooting

• For 2d displays Arc browser is recommended
• For Looking Glass Displays Chrome is recommended

Star History

Contributors