/similarity-search

Primary LanguageJupyter Notebook

Here's a sample README.md for your similarity search project:

Similarity Search with LLaMA 3 and Faiss

This project implements a similarity search system using Flask, Faiss, and LLaMA 3. It allows users to add textual data, search for similar text, and generate context-based responses using the LLaMA 3 API.

Features

  • Add Data: Upload textual data and compute embeddings for similarity search.
  • Search: Find similar texts from the existing data using vector embeddings.
  • Generate Responses: Use the LLaMA 3 API to generate responses based on the most similar text from the search results.

Prerequisites

  • Python 3.x
  • Flask
  • Requests
  • PyTorch
  • Faiss
  • A valid LLaMA 3 API key

Installation

  1. Clone the repository:

    git clone https://github.com/yourusername/similarity-search.git
cd similarity-search

  2. Create a virtual environment and activate it:

    python -m venv venv
source venv/bin/activate  # On Windows use `venv\Scripts\activate`

  3. Install the required packages:

    pip install -r requirements.txt

  4. Set up environment variables:

    • Create a file named .env in the root directory of the project.

    • Add your LLaMA 3 API key to the .env file:

      LLAMA_API_KEY=your_llama_api_key

Usage

  1. Start the Flask server:

    python app.py

  2. Open your web browser and navigate to http://localhost:5000 to access the web interface.

Endpoints

  • GET /: Renders the index page.
  • POST /add: Adds a new text entry to the database. Requires text parameter.
  • POST /search: Searches for similar texts. Requires text parameter.

Code Overview

  • app.py: Main Flask application script.

    • index(): Renders the main page.
    • add_data(): Handles adding new text and saving embeddings.
    • search(): Handles search queries and generates responses using LLaMA 3.
    • get_embedding(): Converts text to embeddings using a model.
    • generate_with_llama(): Interacts with the LLaMA 3 API to generate responses.

  • embeddings.json: Stores text and their embeddings.

Configuration

  • LLAMA_API_URL: URL for the LLaMA 3 API. Change to your LLaMA 3 endpoint.
  • LLAMA_API_KEY: API key for LLaMA 3. Set in the .env file.

Contributing

Feel free to submit issues or pull requests if you have suggestions or improvements.

License

This project is licensed under the MIT License. See the LICENSE file for more details.

Acknowledgements

Replace placeholder URLs and file paths with actual values relevant to your project.