ProfGPT is a virtual reality (VR) educational application developed for Oculus Quest 2. It offers an immersive learning experience for users to explore and experiment with electrical circuits in a virtual lab environment. The highlight of the application is ProfGPT, an AI-powered virtual teacher who guides users through the learning process using natural language interaction (English or French).
- 🔌 Interactive Electrical Circuits Lab: Experiment with a wide range of electrical components to build and explore circuits in a realistic 3D setting.
- 🤖 AI-powered Virtual Instructor: ProfGPT guides users with spoken explanations, utilizing ChatGPT, Whisper, and AWS Polly to answer questions.
- 🎮 Immersive VR Learning: Designed for Oculus Quest 2, offering an engaging and hands-on learning experience in virtual reality.
- 🗣️ Natural Language Assistance: Voice queries are seamlessly processed via Whisper, ChatGPT, and AWS Polly for a smooth interactive experience.
The application architecture is divided into two key components:
- Virtual Lab Table: A space where users can interact with various electrical components to build circuits.
- ProfGPT: An intelligent virtual teacher built using APIs for speech recognition, natural language processing, and text-to-speech services.
The architecture leverages several APIs:
- Whisper (speech-to-text)
- ChatGPT (AI conversational responses)
- AWS Polly (text-to-speech)
ProfGPT uses a modular architecture where each component is responsible for specific interactions:
- Speech Recognition: The user records their voice, which is sent to Whisper by OpenAI for transcription.
- AI Interaction: The transcribed text is sent to ChatGPT to generate a response.
- Speech Synthesis: The AI's response is converted back into speech using AWS Polly and played for the user.
- Unity for the VR environment
- OpenAI's Whisper and ChatGPT for smart voice interaction
- AWS Polly for text-to-speech synthesis
- Whisper: Converts audio input to text using state-of-the-art speech recognition.
- AWS Polly: Provides natural-sounding text-to-speech conversion to generate responses from ProfGPT.
- ChatGPT: AI-powered responses to user queries, allowing natural language interaction.
- ReadyPlayerMe: Used to create the ProfGPT avatar.
- OpenAI-Unity: For integrating OpenAI models (Whisper and ChatGPT) with Unity.
- XR Interaction Toolkit: Facilitates interactions within the VR environment.
- Mixamo: Provides animations for ProfGPT’s avatar.
The virtual lab table was imported from this repo and it contains various components for users to build electrical circuits. Users can interact with components like:
- 💡 Bulbs
- 🔋 Batteries
- 🌞 Solar Panels
These components can be picked up and placed on the lab table to form functioning circuits.
ProfGPT is designed using the ReadyPlayerMe platform, and the avatar is integrated with Unity via the ReadyPlayerMe SDK. The avatar used has the following glb url:
https://models.readyplayer.me/644c3d00b7a1ed40a46033c9.glb
The avatar can:
- Hear user queries using Whisper.
- Respond to users using ChatGPT and AWS Polly.
The following scripts are essential:
ProfFrGPT.cs: Manages interactions between the user and ProfGPT, including sending audio to APIs and handling responses.ProfUI.cs: Manages the GUI for recording voice input and displaying ProfGPT’s responses.
We can see the structure of the scene next to the ProfGPT avatar
ProfGPT next to the lab table. Any question can be asked to ProfGPT and we can see it written in the UI.
This is a side-view of the lab table.
- Faraday Virtual Lab Table - Credit to the creator for providing the virtual lab table asset used in this project.
- OpenAI-Unity GitHub Repo - Credit to the developer for making it easier to integrate OpenAI's models with Unity.
- Sgt3v YouTube Channel - Special thanks to Sgt3v for providing invaluable tutorials on developing smart VR NPCs powered by GPT. Check out his channel for amazing content!
This project is incomplete. The repository contains all the files I could retrieve from the original project. Due to the age of the project, some assets and code might be missing or outdated. Please take this into consideration when reviewing or attempting to use the code.