NVIDIA Generative AI Examples

Introduction

State-of-the-art Generative AI examples that are easy to deploy, test, and extend. All examples run on the high performance NVIDIA CUDA-X software stack and NVIDIA GPUs.

NVIDIA NGC

Generative AI Examples uses resources from the NVIDIA NGC AI Development Catalog.

GPU-optimized containers used in these examples
Release notes and developer documentation

Retrieval Augmented Generation (RAG)

A RAG pipeline embeds multimodal data -- such as documents, images, and video -- into a database connected to a LLM. RAG lets users chat with their data!

Developer RAG Examples

The developer RAG examples run on a single VM. They demonstrate how to combine NVIDIA GPU acceleration with popular LLM programming frameworks using NVIDIA's open source connectors. The examples are easy to deploy via Docker Compose.

Examples support local and remote inference endpoints. If you have a GPU, you can inference locally via TensorRT-LLM. If you don't have a GPU, you can inference and embed remotely via NVIDIA AI Foundations endpoints.

Model	Embedding	Framework	Description	Multi-GPU	TRT-LLM	NVIDIA AI Foundation	Triton	Vector Database
llama-2	e5-large-v2	Llamaindex	Canonical QA Chatbot	YES	YES	No	YES	Milvus/PGVector
mixtral_8x7b	nvolveqa_40k	Langchain	Nvidia AI foundation based QA Chatbot	No	No	YES	YES	FAISS
llama-2	all-MiniLM-L6-v2	Llama Index	QA Chatbot, GeForce, Windows	NO	YES	NO	NO	FAISS
llama-2	nvolveqa_40k	Langchain	QA Chatbot, Task Decomposition Agent	No	No	YES	YES	FAISS
mixtral_8x7b	nvolveqa_40k	Langchain	Minimilastic example showcasing RAG using Nvidia AI foundation models	No	No	YES	YES	FAISS

Enterprise RAG Examples

The enterprise RAG examples run as microservies distributed across multiple VMs and GPUs. They show how RAG pipelines can be orchestrated with Kubernetes and deployed with Helm.

Enterprise RAG examples include a Kubernetes operator for LLM lifecycle management. It is compatible with the NVIDIA GPU operator that automates GPU discovery and lifecycle management in a Kubernetes cluster.

Enterprise RAG examples also support local and remote inference via TensorRT-LLM and NVIDIA AI Foundations endpoints.

Model	Embedding	Framework	Description	Multi-GPU	Multi-node	TRT-LLM	NVIDIA AI Foundation	Triton	Vector Database
llama-2	NV-Embed-QA-003	Llamaindex	QA Chatbot, Helm, k8s	NO	NO	YES	NO	YES	Milvus

Tools

Example tools and tutorials to enhance LLM development and productivity when using NVIDIA RAG pipelines.

Name	Description	Deployment	Tutorial
Evaluation	Example open source RAG eval tool that uses synthetic data generation and LLM-as-a-judge	Docker compose file	README
Observability	Observability serves as an efficient mechanism for both monitoring and debugging RAG pipelines.	Docker compose file	README

Open Source Integrations

These are open source connectors for NVIDIA-hosted and self-hosted API endpoints. These open source connectors are maintained and tested by NVIDIA engineers.

Name	Framework	Chat	Text Embedding	Python	Description
NVIDIA AI Foundation Endpoints	Langchain	YES	YES	YES	Easy access to NVIDIA hosted models. Supports chat, embedding, code generation, steerLM, multimodal, and RAG.
NVIDIA Triton + TensorRT-LLM	Langchain	YES	YES	YES	This connector allows Langchain to remotely interact with a Triton inference server over GRPC or HTTP tfor optimized LLM inference.
NVIDIA Triton Inference Server	LlamaIndex	YES	YES	NO	Triton inference server provides API access to hosted LLM models over gRPC.
NVIDIA TensorRT-LLM	LlamaIndex	YES	YES	NO	TensorRT-LLM provides a Python API to build TensorRT engines with state-of-the-art optimizations for LLM inference on NVIDIA GPUs.

NVIDIA support

In each example README we indicate the level of support provided.

Feedback / Contributions

We're posting these examples on GitHub to support the NVIDIA LLM community, facilitate feedback. We invite contributions via GitHub Issues or pull requests!

Known issues

In each of the READMEs, we indicate any known issues and encourage the community to provide feedback.
The datasets provided as part of this project is under a different license for research and evaluation purposes.
This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.

jtmancilla/GenerativeAIExamples