aka Emerging Architectures for LLM Applications
This is a list of available tools, projects, and vendors at each layer of the LLM app stack.
Our original article included only the most popular options, based on user interviews. This repo is meant to be more comprehensive, covering all available options in each category. We probably still missed some important projects, so please open a PR if you see anything missing.
We also included Perplexity and Cursor.sh prompts to make searching and markdown table formatting easier.
Data Pipelines Embedding Models Vector Databases Playgrounds Orchestrators APIs / Plugins LLM Caches Logging / Monitoring / Eval Validators LLM APIs (proprietary) LLM APIs (open source) App Hosting Platforms Cloud Providers Opinionated Clouds
Name (site)
Description
Github
Pip Installs
Databricks A unified data platform for building, deploying, and maintaining enterprise data solutions, including products (like MosaicML and MLflow) purpose-built for AI
Airflow A data pipeline framework to programmatically author, schedule, and monitor data pipelines and workflows, including for LLMs
Unstructured.io Open-source components for pre-processing documents such as PDFs, HTML, and Word documents for usage with LLM apps
Fivetran A platform that extracts, loads, and transforms data from various sources for analytics, AI, and operations
N/A
Airbyte An open-source data integration engine that helps consolidate data in data warehouses, lakes, and databases
^ Back to Contents ^
Name (site)
Description
Github
Pip Installs
OpenAI Ada Embedding 2 OpenAI's most popular embedding model for capturing semantic relationships in text
n/a
Cohere AI An independent commerical provider of LLMs, with particular focus on embeddings for semantic search, topic clustering, and vertical applications
Sentence Transformers An open-source Python framework for sentence, text, and image embeddings
^ Back to Contents ^
Name (site)
Description
Github
Pip Installs
Pinecone A managed, cloud-native vector database with a simple API for high-performance AI applications
n/a
Weaviate An open-source vector database that stores both objects and vectors
ChromaDB An AI-native, open-source embedding database platform for developers
Pgvector An open-source vector similarity search for Postgres, allowing for exact and approximate nearest-neighbor search
Zilliz (Milvus) An open-source vector database, built for developing and maintaining AI applications
Qdrant A vector database and vector similarity search engine
Metal io A managed service for developers to build applications with ML embeddings
N/A
LanceDB A serverless vector database for AI applications
^ Back to Contents ^
Name (site)
Description
Github
Pip Installs
OpenAI Playground A web-based platform for experimenting with various machine-learning models developed by OpenAI
N/A
N/A
nat.dev A platform that allows users to test prompts with multiple language models and compare their performance
Humanloop A platform that helps developers build applications on top of LLMs
^ Back to Contents ^
Name (site)
Description
Github
Pip Installs
Langchain An open-source library that gives developers the tools to build applications powered by LLMs
LlamaIndex A data framework for LLM applications to ingest, structure, and access private or domain-specific data
Autogen A framework for automating and streamlining LLM workflows using customizable, conversable agents for complex AI applications
Microsoft Semantic Kernel A lightweight open-source orchestration SDK
Vercel AI SDK An open-source library for developers to build streaming UIs in JavaScript and TypeScript
Vectara AI A search and discovery platform for AI conversations utilizing your own data
N/A
ChatGPT An AI chatbot that uses natural language processing to create humanlike conversational dialogue
N/A
N/A
^ Back to Contents ^
Name (site)
Description
Github
Pip Installs
Serp API A real-time API to access Google search results, as well as handling proxies, solving captchas, and parsing structured data
Wolfram Alpha API A web-based API providing computational and presentation capabilities for integration into various applications
N/A
Zapier API AI Plugin A plugin that allows you to connect 5,000+ apps and interact with them directly inside ChatGPT
N/A
N/A
^ Back to Contents ^
Name (site)
Description
Github
Pip Installs
Redis An in-memory data structure store used as a database, cache, message broker, and streaming engine
SQLite A self-contained, serverless, zero-configuration, transactional SQL database engine
GPTCache An open-source tool for improving the efficiency and speed of GPT-based applications by implementing a cache to store the responses
N/A
^ Back to Contents ^
Logging / Monitoring / Eval
Name (site)
Description
Github
Pip Installs
Braintrust Data An AI product stack featuring evaluations, prompt playgrounds, continuous integration, dataset management, and access to various AI models through a single API
Arize AI An observability platform for both LLMs and supervised ML
Weights & Biases An MLOps platform for streamlining ML workflows
MLflow A platform to streamline ML development
PromptLayer A platform for tracking, managing, and sharing LLM prompt engineering
Helicone An open-source observability platform for LLMs
Portkey AI A platform to develop, launch, maintain, and iterate generative AI apps and features
N/A
N/A
Freeplay AI A platform to prototype, test, and optimize LLM features for customers
N/A
N/A
Gentrace An API and SDKs for evaluating and observing generative data, with features like AI, heuristic, and human grading evaluations, as well as production data observation
N/A
Patronus AI An automated evaluation and benchmarking platform for LLMs, providing tools for testing, scoring, and evaluating LLMs in real-world scenarios
N/A
N/A
Autoblocks AI A collaborative cloud-based workspace designed for rapid iteration on GenAI products, offering features like prompt management, observability, continuous evaluations, fine-tuning, prototyping, debugging, and scalable data ingestion & search, all in a provider-agnostic environment
N/A
Context AI Tools for pre-launch LLM evaluations and post-launch analytics, with features such as testing, performance monitoring, user conversation analysis, and support for various models and libraries
N/A
E2b dev Services to deploy, test, and monitor AI agents, including a sandbox with a secure, long-running cloud environment for various LLMs with features like internet access
Agentops Toolkit for evaluating and developing AI agents, providing tools for agent development, monitoring capabilities, and replay analytics
Zenoml AI evaluation platform that enables data visualization, model performance analysis, and the creation of interactive reports for various data types
Baserun Tools for model configuration, prompt playground, monitoring, and prototype workflow, as well as features for full visibility into LLM workflows and end-to-end testing
^ Back to Contents ^
Name (site)
Description
Github
Pip Installs
Guardrails AI An open-source Python package for specifying structure and type, validating, and correcting the outputs of LLMs
Rebuff An open-source framework designed to detect and protect against prompt injection attacks in LLM apps
Microsoft Guidance A guidance language for controlling LLMs, providing a syntax for architecting LLM workflows
LMQL An open-source programming language and platform for language model interaction
Outlines A tool for helping developers guide text generation to build robust interfaces with external systems and guarantee that outputs match a regex or JSON schema
^ Back to Contents ^
Name (site)
Description
Github
Pip Installs
OpenAI A company providing many leading LLMs, including the GPT-3.5 and GPT-4 families
N/A
Anthropic The developer of Claude, an AI assistant based on Anthropic’s research
N/A
Cohere AI An LLM vendor with particular focus on embeddings for semantic search, topic clustering, and vertical applications
^ Back to Contents ^
Name (site)
Description
Github
Pip Installs
Hugging Face A hub for open-source AI models and inference endpoints, including leading base LLMs and LoRAs/fine-tunes
Replicate An AI hosting platform and model inference hub that allows software developers to integrate AI models into their apps
^ Back to Contents ^
Name (site)
Description
Vercel A cloud platform designed for front-end engineers, built with first-class support for LLM apps
Netlify An enterprise cloud computing company that offers a development platform for web applications and dynamic websites
Steamship An SDK and hosting platform for AI agents and tools, both a package manager and package hosting service for AI
Streamlit An open-source Python library designed for creating and sharing custom web apps for ML and data science
Modal A platform that enables running distributed applications using the modal Python package
^ Back to Contents ^
Name (site)
Description
Amazon Web Services A cloud computing platform, offering services from data centers globally
Google Cloud Platform A cloud computing platform, offering services from data centers globally
Microsoft Azure A cloud computing platform, offering services from data centers globally
CoreWeave A specialized cloud provider that delivers GPUs on top of flexible deployment infrastructure
^ Back to Contents ^
Name (site)
Description
Databricks (MosaicML) Databricks acquired Mosaic ML in 2023, along with its tooling and platform for efficient pre-trainining, fine-tuning and inferencing LLMs
Anyscale A fully managed compute platform that enables the development, deployment, and management of Ray (Python) apps
Modal A platform that eables running distributed applications using the Modal Python package
Runpod A cloud computing platform designed for AI and ML applications
OctoML A compute service that allows users to run, tune, and scale generative models
^ Back to Contents ^
Formatting Prompt Templates We were able to partialy automate this - particularly finding Github and PyPI links - using this Perplexity search prompt . It worked roughly ~75% of the time and could handle ~3 projects at a time, pulling data from 20-30 sources in each iteration.
Once you have the data you would like to add, if you don't want deal with the markdown formatting here, it is easy to correctly format using a tool like Cursor .
See the prompt below that works as an inline edit, just make sure you highlight 4-5 previous examples so Cursor can infer the format itself:
^ Back to Contents ^
(node/npm)
