AI-Powered Knowledge Base

Turn.any.website.into.an.llm-powered.knowledge.base.mp4

This repository contains an AI-powered knowledge base that utilizes the LLMs model to answer questions based on a given website's content and provide sources as links to the relevant pages.

The system:

Loads the website's content using a sitemap
Split each web page into chunks
Embed each chunk using a LLM (for now OpenAI) and store them in the **Chroma vector database
Then it embeds the user query and run a similarity search using the Chroma database
Finally it loads the similarity search results as context for a LLM (for now ChatGPT) to find relevant answers and citing the sources

It also provides a Streamlit-based web interface for an easy-to-use experience.

Files

knowledge_base.py: The main module that creates the KnowledgeBase class. This class is responsible for loading and processing the website content, creating the document index, and querying the LLM model for answers.
app.py: A Streamlit web application that provides a user interface for querying the AI-powered knowledge base.

Installation

Clone the repository:

git clone git@github.com:bitswired/website-to-knowledge-base.git

Instal the project with poetry:

poetry install

Usage

Knowledge Base

To use the KnowledgeBase class, follow these steps:

Import the KnowledgeBase class:

from knowledge_base import KnowledgeBase

Instantiate the KnowledgeBase with the appropriate sitemap URL and pattern (optional):

kb = KnowledgeBase(
    sitemap_url="https://nextjs.org/sitemap.xml",
    pattern="docs/api-refe",
    chunk_size=8000,
    chunk_overlap=3000,
)

Ask a question:

result = kb.ask("How do I deploy my Next.js app?")
print(result)

Web Application

To run the Streamlit web application, execute the following command in your terminal:

streamlit run app.py

The web app will open in your default browser. Enter the URL to the website's sitemap, an optional filter pattern for the URLs, and your question. The AI-powered knowledge base will return an answer based on the content of the website.

Requirements

An API key for OpenAI's GPT-4 (see OpenAI's API documentation for details)

License

This project is licensed under the MIT License.

zorost/website-to-knowledge-base