/Sports-Buddy

Conversational AI with GPT-4 Vision, OpenAI Whisper, and TTS

Primary LanguagePythonMIT LicenseMIT

Conversational AI with GPT-4 Vision, OpenAI Whisper, and TTS

Overview

This project integrates GPT-4 Vision, OpenAI Whisper, and OpenAI Text-to-Speech (TTS) to create an interactive AI system for conversations. It combines visual and audio inputs for a seamless user experience.

Demo Video:

https://twitter.com/ayushspai/status/1726222559480557647

Components

  • GPT-4 Vision: Analyzes visual input and generates contextual responses.
  • OpenAI Whisper: Converts spoken language into text.
  • OpenAI TTS: Transforms text responses into spoken language.

Main Files

  • main.py: Manages audio processing, image encoding, AI interactions, and text-to-speech output.
  • capture.py: Captures and processes video frames for visual analysis.

Installation

Prerequisites

  • Python 3.x
  • An OpenAI API key (set as an environment variable OPENAI_API_KEY)

Libraries

Install the necessary libraries with the requirements.txt file.

pip install -r requirements.txt

Usage

Running the Scripts

  • Start capture.py: Captures video frames and saves them for AI analysis.

    • Reads a video file, displays the video, and saves the current frame as frame.jpg.
    • Execute with python capture.py.

  • Run main.py concurrently: Orchestrates the conversational AI.

    • Continuously listens for user audio input.
    • Transcribes speech to text, captures the current video frame, and sends both to GPT-4 for analysis.
    • Converts the AI's response to speech and plays it back.
    • Execute with python main.py.

Workflow

  1. main.py listens for audio input and transcribes it using OpenAI Whisper.
  2. Meanwhile, capture.py captures a video frame.
  3. Both the audio transcription and the encoded image are sent to GPT-4 Vision.
  4. GPT-4 Vision responds, considering the visual and textual context.
  5. The response is vocalized using OpenAI TTS and played to the user.

Notes

  • Ensure both main.py and capture.py are active for the system to function.
  • The video file in capture.py can be customized.
  • Adequate hardware is recommended for smooth audio and video processing.

Conclusion

This project demonstrates a novel approach to combining various AI technologies, creating a dynamic and interactive conversational AI experience. It harnesses the capabilities of GPT-4 Vision, Whisper, and TTS for a comprehensive audio-visual interaction.