/LlmTestApp

React / NET 8 app for searching documents with a LLM. Uses ChromaDB, LlamaSharp, AllMiniLmL6v2Sharp, and ChromaDBSharp. No Python!

Primary LanguageC#MIT LicenseMIT

React GPT test app

React/ASP Net 8 App for Retrieval-Augemented Generation on local hardware (no apis). Uses C# for embedding generation via AllMiniLML6v2Sharp, LLAMA-2-Chat-7b via LLamaSharp, and connection to a ChromaDB with ChromaDBSharp all running on the local machine.

This app is for demo purposes only, to show the whole pipeline can be run from .NET, and not for production use.

How to run.

  • Start a hosted version of chromadb. Follow instructions here. I recommend using the docker build for an easy solution.

  • Download the All-Mini-LM-L6-v2 .onnx and .vocab from here

  • Download the .gguf LLAMA weights. For example LLama 2 Chat 7b

  • Update appsettings.json to point to models/chromadb

{
    "AppSettings": {
        "ChromaDbUrl": "http://chroma-endpoint",
        "ChromaDocumentCollection": "chroma-collection-name",
        "Separators": [ "\n\n", "\n", " ", "" ],
        "AllMiniV2Vocab": "path/to/vocab.txt",
        "AllMiniV2Model": "path/to/model.onnx",
        "ChatModelPath": "path/to/weights.gguf"
    }
}
  • Run the app. dotnet build then dotnet run or run with visual studio.

Alt text

  • Add some documents. (Currently only supports pasting text.)

Alt text

  • Find answers via Vector Search.

Alt text

  • Find answers via LLM

Alt text