/frogbase

Transform audio-visual content into navigable knowledge.

Primary LanguagePythonMIT LicenseMIT

🐸 FrogBase

Create navigable knowledge from multi-media content

FrogBase (previously whisper-ui) simplifies the download-transcribe-embed-index workflow for multi-media content. It does so by linking content from various platforms (yt_dlp) with speech-to-text models (OpenAI's Whisper), image & text encoders (SentenceTransformers), and embedding stores (hnswlib).

⚠️ Warning: This is currently a pre-release version and is known to be very unstable. For stable releases, please use any 1.x versions.

from frogbase import FrogBase
fb = FrogBase()
fb.demo()
fb.search("What is the name of the squeaky frog?")

Full Documentation (WIP).

FrogBase also comes with a ready-to-use UI for non-technical users!

whisper-ui-update-demo.mp4

PyPI Status Python Version License

Features

FrogBase currently provides functionality to:

  • Download media files from a wide range of platforms (YouTube, TikTok, Vimeo, etc.) using yt_dlp
  • Transcribe audio streams for downloaded & local files using OpenAI's Whisper
  • Embed transcribed text from corresponding video segments using Sentence Transformers
  • Index & search the embedded content using hnswlib

FrogBase also includes a Streamlit UI to provide a simple GUI for the above functionality enabling a locally hosted, interactive experience.

Quickstart

Software Developers

This section is for software developers who want to use FrogBase as a python package.

  1. Install ffmpeg and FrogBase

    sudo apt install ffmpeg
pip install frogbase

  2. Import FrogBase and use it as follows -

    from frogbase import FrogBase

fb = FrogBase()

sources = [
   "https://www.youtube.com/watch?v=HBxn56l9WcU",
   "https://www.youtube.com/@hayabhay"
]

fb.add(sources)

fb.search("What is the name of the squeaky frog?")

Non-technical Users

This section is for non-technical users who want to use FrogBase primarily through the accompanying Streamlit UI.

  1. Download the latest release of FrogBase from here and unzip it. Or, you can also clone the repository console git clone https://github.com/hayabhay/frogbase.git

  2. Install FrogBase dependencies manually and run the UI.

    Note: This also requires ffmpeg to be installed on your system. You can install it using sudo apt install ffmpeg on Ubuntu.

    1. Using pip

      pip install frogbase streamlit
streamlit run ui/01_🏠_Home.py

[Coming soon] Instructions, environment for installation using Docker & Anaconda

Links