/lightspeed-rag-pipeline

lightspeed rag pipeline

Primary LanguagePython

LightSpeed RAG tetkon pipeline

This repo contains Tekton pipeline for creation of LLM embedding using Llama_index

  • Loading data
  • Indexing
  • Storing
  • Evaluating

Prerequisites

  • Tekton
  • OpenShift
  • Nvidia Operator installed
  • Kostomize CLI

Usage

  • Build the embedding container under images/rag using the build.sh file.
  • Add secrets resources to the secret folder and to link the secretes to the pipeline service account
  • Copy and edit the pipeline configuration file pipeline/data-pipeline.yaml
  • add you file adn resources to th kustomization.yaml file
  • Deploy the pipeline pipeline using the following command:
oc apply -k .
  • Run the pipeline
  • Results:
    • Container image in the specified location with the embedding json files
    • GitHub Release zip file which contains the embedding files and metadata files

Folders

  • images - base images containing the tekton tasks

References

https://docs.llamaindex.ai/en/stable/getting_started/concepts.html https://docs.llamaindex.ai/en/stable/understanding/using_llms/using_llms.htm https://blog.llamaindex.ai/build-and-scale-a-powerful-query-engine-with-llamaindex-and-ray-bfb456404bc4 https://tekton.dev/docs/pipelines/pipelines/