/real_time_rag_kafka

rag with kafka pipelines

Primary LanguagePython

Real Time RAG using Kafka pipelines

This project shows how to create a real time kafka pipeline for RAG.

Project Overview

This project is designed with a producer and consumer setup for real time RAG using kakfa pipelines using Vlite as a numpy vector database.

How to run

  1. Install the dependencies

pip install requirements.txt

  1. Start Zookeeper

.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties

  1. Start Kafka Server

.\bin\windows\kafka-server-start.bat .\config\server.properties

  1. Create a topic

.\bin\windows\kafka-topics.bat --create --topic myfirsttopic --bootstrap-server localhost:9092

  1. Start the producer

python producer.py

  1. Start the consumer

python consumer.py