[Oct'23] DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
[Dec'23] DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines
[Dec'22] Demonstrate-Search-Predict: Composing Retrieval & Language Models for Knowledge-Intensive NLP
Getting Started:
DSPy is a framework for developing and optimizing LM pipelines for complex tasks. To use LMs to build a complex system without DSPy, you generally have to: (1) break the problem down into steps, (2) prompt your LM well for each step, (3) generate synthetic data to tune each step, and (4) use that data to finetune smaller LMs to cut costs.
Composing these into the right multi-step pipelines is a hard, messy process. DSPy creates a systematic space of modular and trainable pieces, instead of hacky 'prompt engineering' or one-off pipeline tuning. To do that, DSPy separates the flow of your program (modules) from the parameters (LM prompts and weights) of each step. DSPy introduces new optimizers, which are algorithms that can craft and tune the parameters of your pipeline, given a metric you want to maximize.
Using DSPy is an iterative process. You first define your task and the metrics you want to maximize, and prepare a few example inputs — typically without labels (or only with labels for the final outputs, if your metric requires them). Then, you build your pipeline by selecting built-in layers (modules) to use, giving each layer a signature (input/output spec), and then calling your modules freely in your Python code. Lastly, you use a DSPy optimizer to compile your code into high-quality instructions, automatic few-shot examples, or updated LM weights for your LM.
DSPy can routinely teach powerful models like GPT-3.5 or GPT-4 and local models like T5-base or Llama2-13b to be much more reliable at tasks. DSPy will compile the same program into different few-shot prompts and/or finetunes for each LM.
- Installation
- Tutorials & Documentation
- Framework Syntax
- Compiling: Two Powerful Concepts
- FAQ: Is DSPy right for me?
When we build neural networks, we don't write manual for-loops over lists of hand-tuned floats. Instead, you might use a framework like PyTorch to compose declarative layers (e.g., Convolution or Dropout) and then use optimizers (e.g., SGD or Adam) to learn the parameters of the network.
Ditto! DSPy gives you the right general-purpose modules (e.g., ChainOfThought, ReAct, etc.), which replace string-based prompting tricks. To replace prompt hacking and one-off synthetic data generators, DSPy also gives you general optimizers (BootstrapFewShotWithRandomSearch or BayesianSignatureOptimizer), which are algorithms that update parameters in your program. Whenever you modify your code, your data, you assertions, or your metric, you can compile your program again and DSPy will create new effective prompts that fit your changes.
All you need is:
pip install dspy-ai
Or open our intro notebook in Google Colab:
By default, DSPy depends on openai==0.28. However, if you install openai>=1.0, the library will use that just fine. Both are supported.
For the optional Pinecone, Qdrant, chromadb, or marqo retrieval integration(s), include the extra(s) below:
pip install dspy-ai[pinecone] # or [qdrant] or [chromadb] or [marqo] or [mongodb]
The DSPy documentation is divided into tutorials (step-by-step illustration of solving a task in DSPy), guides (how to use specific parts of the API), and examples (self-contained programs that illustrate usage).
| Level | Tutorial | Run in Colab | Description |
|---|---|---|---|
| Beginner | Getting Started |  |
Introduces the basic building blocks in DSPy. Tackles the task of complex question answering with HotPotQA. |
| Beginner | Compiling for Tricky Tasks | N/A | Teaches LMs to reason about logical statements and negation. Uses GPT-4 to bootstrap few-shot CoT demonstations for GPT-3.5. Establishes a state-of-the-art result on ScoNe. Contributed by Chris Potts. |
| Beginner | Local Models & Custom Datasets | |
Illustrates two different things together: how to use local models (Llama-2-13B in particular) and how to use your own data examples for training and development. |
| Intermediate | The DSPy Paper | N/A | Sections 3, 5, 6, and 7 of the DSPy paper can be consumed as a tutorial. They include explained code snippets, results, and discussions of the abstractions and API. |
| Intermediate | Finetuning for Complex Programs | |
Teaches a local T5 model (770M) to do exceptionally well on HotPotQA. Uses only 200 labeled answers. Uses no hand-written prompts, no calls to OpenAI, and no labels for retrieval or reasoning. |
| Advanced | Information Extraction | |
Tackles extracting information from long articles (biomedical research papers). Combines in-context learning and retrieval to set SOTA on BioDEX. Contributed by Karel D’Oosterlinck. |
You may also find this YouTube intro useful. Many people have told us they "got" the framework best after watching.
If you're new to DSPy, it's probably best to go in sequential order. You will probably refer to these guides frequently after that, e.g. to copy/paste snippets that you can edit for your own DSPy programs.
The DSPy team believes complexity has to be justified. We take this seriously: we never release a complex tutorial (above) or example (below) unless we can demonstrate empirically that this complexity has generally led to improved quality or cost. This kind of rule is rarely enforced by other frameworks or docs, but you can count on it in DSPy examples.
There's a bunch of examples in the examples/ directory and in the top-level directory. We welcome contributions!
You can find other examples tweeted by @lateinteraction on Twitter/X.
DSPy hides tedious prompt engineering, but it cleanly exposes the important decisions you need to make: [1] what's your system design going to look like? [2] what are the important constraints on the behavior of your program?
You express your system as free-form Pythonic modules. DSPy will tune the quality of your program in whatever way you use foundation models: you can code with loops, if statements, or exceptions, and use DSPy modules within any Python control flow you think works for your task.
Suppose you want to build a simple retrieval-augmented generation (RAG) system for question answering. You can define your own RAG program like this:
class RAG(dspy.Module):
def __init__(self, num_passages=3):
super().__init__()
self.retrieve = dspy.Retrieve(k=num_passages)
self.generate_answer = dspy.ChainOfThought("context, question -> answer")
def forward(self, question):
context = self.retrieve(question).passages
answer = self.generate_answer(context=context, question=question)
return answerA program has two key methods, which you can edit to fit your needs.
Your __init__ method declares the modules you will use. Here, RAG will use the built-in Retrieve for retrieval and ChainOfThought for generating answers. DSPy offers general-purpose modules that take the shape of your own sub-tasks — and not pre-built functions for specific applications.
Modules that use the LM, like ChainOfThought, require a signature. That is a declarative spec that tells the module what it's expected to do. In this example, we use the short-hand signature notation context, question -> answer to tell ChainOfThought it will be given some context and a question and must produce an answer. We will discuss more advanced signatures below.
Your forward method expresses any computation you want to do with your modules. In this case, we use the modules self.retrieve and self.generate_answer to search for some context and then use the context and question to generate the answer!
You can now either use this RAG program in zero-shot mode. Or compile it to obtain higher quality. Zero-shot usage is simple. Just define an instance of your program and then call it:
rag = RAG() # zero-shot, uncompiled version of RAG
rag("what is the capital of France?").answer # -> "Paris"The next section will discuss how to compile our simple RAG program. When we compile it, the DSPy compiler will annotate demonstrations of its steps: (1) retrieval, (2) using context, and (3) using chain-of-thought to answer questions. From these demonstrations, the DSPy compiler will make sure it produces an effective few-shot prompt that works well with your LM, retrieval model, and data. If you're working with small models, it'll finetune your model (instead of prompting) to do this task.
If you later decide you need another step in your pipeline, just add another module and compile again. Maybe add a module that takes the chat history into account during search?
To make it possible to compile any program you write, DSPy introduces two simple concepts: Signatures and Teleprompters.
When we assign tasks to LMs in DSPy, we specify the behavior we need as a Signature. A signature is a declarative specification of input/output behavior of a DSPy module.
Instead of investing effort into how to get your LM to do a sub-task, signatures enable you to inform DSPy what the sub-task is. Later, the DSPy compiler will figure out how to build a complex prompt for your large LM (or finetune your small LM) specifically for your signature, on your data, and within your pipeline.
A signature consists of three simple elements:
- A minimal description of the sub-task the LM is supposed to solve.
- A description of one or more input fields (e.g., input question) that will we will give to the LM.
- A description of one or more output fields (e.g., the question's answer) that we will expect from the LM.
We support two notations for expressing signatures. The short-hand signature notation is for quick development. You just provide your module (e.g., dspy.ChainOfThought) with a string with input_field_name_1, ... -> output_field_name_1, ... with the fields separated by commas.
In the RAG class earlier, we saw:
self.generate_answer = dspy.ChainOfThought("context, question -> answer")In many cases, this barebones signature is sufficient. However, sometimes you need more control. In these cases, we can use the full notation to express a more fully-fledged signature below.
class GenerateSearchQuery(dspy.Signature):
"""Write a simple search query that will help answer a complex question."""
context = dspy.InputField(desc="may contain relevant facts")
question = dspy.InputField()
query = dspy.OutputField()
### inside your program's __init__ function
self.generate_answer = dspy.ChainOfThought(GenerateSearchQuery)You can optionally provide a prefix and/or desc key for each input or output field to refine or constraint the behavior of modules using your signature. The description of the sub-task itself is specified as the docstring (i.e., """Write a simple...""").
After defining the RAG program, we can compile it. Compiling a program will update the parameters stored in each module. For large LMs, this is primarily in the form of creating and validating good demonstrations for inclusion in your prompt(s).
Compiling depends on three things: a (potentially tiny) training set, a metric for validation, and your choice of teleprompter from DSPy. Teleprompters are powerful optimizers (included in DSPy) that can learn to bootstrap and select effective prompts for the modules of any program. (The "tele-" in the name means "at a distance", i.e., automatic prompting at a distance.)
DSPy typically requires very minimal labeling. For example, our RAG pipeline may work well with just a handful of examples that contain a question and its (human-annotated) answer. Your pipeline may involve multiple complex steps: our basic RAG example includes a retrieved context, a chain of thought, and the answer. However, you only need labels for the initial question and the final answer. DSPy will bootstrap any intermediate labels needed to support your pipeline. If you change your pipeline in any way, the data bootstrapped will change accordingly!
my_rag_trainset = [
dspy.Example(
question="Which award did Gary Zukav's first book receive?",
answer="National Book Award"
),
...
]Second, define your validation logic, which will express some constraints on the behavior of your program or individual modules. For RAG, we might express a simple check like this:
def validate_context_and_answer(example, pred, trace=None):
# check the gold label and the predicted answer are the same
answer_match = example.answer.lower() == pred.answer.lower()
# check the predicted answer comes from one of the retrieved contexts
context_match = any((pred.answer.lower() in c) for c in pred.context)
return answer_match and context_matchDifferent teleprompters offer various tradeoffs in terms of how much they optimize cost versus quality, etc. For RAG, we might use the simple teleprompter called BootstrapFewShot. To do so, we instantiate the teleprompter itself with a validation function my_rag_validation_logic and then compile against some training set my_rag_trainset.
from dspy.teleprompt import BootstrapFewShot
teleprompter = BootstrapFewShot(metric=my_rag_validation_logic)
compiled_rag = teleprompter.compile(RAG(), trainset=my_rag_trainset)If we now use compiled_rag, it will invoke our LM with rich prompts with few-shot demonstrations of chain-of-thought retrieval-augmented question answering on our data.
The DSPy philosophy and abstraction differ significantly from other libraries and frameworks, so it's usually straightforward to decide when DSPy is (or isn't) the right framework for your usecase.
If you're a NLP/AI researcher (or a practitioner exploring new pipelines or new tasks), the answer is generally an invariable yes. If you're a practitioner doing other things, please read on.
In other words: Why can't I just write my prompts directly as string templates? Well, for extremely simple settings, this might work just fine. (If you're familiar with neural networks, this is like expressing a tiny two-layer NN as a Python for-loop. It kinda works.)
However, when you need higher quality (or manageable cost), then you need to iteratively explore multi-stage decomposition, improved prompting, data bootstrapping, careful finetuning, retrieval augmentation, and/or using smaller (or cheaper, or local) models. The true expressive power of building with foundation models lies in the interactions between these pieces. But every time you change one piece, you likely break (or weaken) multiple other components.
DSPy cleanly abstracts away (and powerfully optimizes) the parts of these interactions that are external to your actual system design. It lets you focus on designing the module-level interactions: the same program expressed in 10 or 20 lines of DSPy can easily be compiled into multi-stage instructions for GPT-4, detailed prompts for Llama2-13b, or finetunes for T5-base.
Oh, and you wouldn't need to maintain long, brittle, model-specific strings at the core of your project anymore.
Note: If you use LangChain as a thin wrapper around your own prompt strings, refer to answer [5.a] instead.
LangChain and LlamaIndex are popular libraries that target high-level application development with LMs. They offer many batteries-included, pre-built application modules that plug in with your data or configuration. In practice, indeed, many usecases genuinely don't need any special components. If you'd be happy to use someone's generic, off-the-shelf prompt for question answering over PDFs or standard text-to-SQL as long as it's easy to set up on your data, then you will probably find a very rich ecosystem in these libraries.
Unlike these libraries, DSPy doesn't internally contain hand-crafted prompts that target specific applications you can build. Instead, DSPy introduces a very small set of much more powerful and general-purpose modules that can learn to prompt (or finetune) your LM within your pipeline on your data.
DSPy offers a whole different degree of modularity: when you change your data, make tweaks to your program's control flow, or change your target LM, the DSPy compiler can map your program into a new set of prompts (or finetunes) that are optimized specifically for this pipeline. Because of this, you may find that DSPy obtains the highest quality for your task, with the least effort, provided you're willing to implement (or extend) your own short program. In short, DSPy is for when you need a lightweight but automatically-optimizing programming model — not a library of predefined prompts and integrations.
If you're familiar with neural networks:
This is like the difference between PyTorch (i.e., representing DSPy) and HuggingFace Transformers (i.e., representing the higher-level libraries). If you simply want to use off-the-shelf
BERT-base-uncasedorGPT2-largeor apply minimal finetuning to them, HF Transformers makes it very straightforward. If, however, you're looking to build your own architecture (or extend an existing one significantly), you have to quickly drop down into something much more modular like PyTorch. Luckily, HF Transformers is implemented in backends like PyTorch. We are similarly excited about high-level wrapper around DSPy for common applications. If this is implemented using DSPy, your high-level application can also adapt significantly to your data in a way that static prompt chains won't. Please open an issue if this is something you want to help with.
Guidance, LMQL, RELM, and Outlines are all exciting new libraries for controlling the individual completions of LMs, e.g., if you want to enforce JSON output schema or constrain sampling to a particular regular expression.
This is very useful in many settings, but it's generally focused on low-level, structured control of a single LM call. It doesn't help ensure the JSON (or structured output) you get is going to be correct or useful for your task.
In contrast, DSPy automatically optimizes the prompts in your programs to align them with various task needs, which may also include producing valid structured ouputs. That said, we are considering allowing Signatures in DSPy to express regex-like constraints that are implemented by these libraries.
DSPy is led by Omar Khattab at Stanford NLP with Chris Potts and Matei Zaharia.
Key contributors and team members include Arnav Singhvi, Paridhi Maheshwari, Keshav Santhanam, Sri Vardhamanan, Eric Zhang, Hanna Moazam, Thomas Joshi, Saiful Haq, and Ashutosh Sharma.
DSPy includes important contributions from Rick Battle and Igor Kotenkov. It reflects discussions with Lisa Li, David Hall, Ashwin Paranjape, Heather Miller, Chris Manning, Percy Liang, and many others.
The DSPy logo is designed by Chuyi Zhang.
To stay up to date or learn more, follow @lateinteraction on Twitter.
If you use DSPy or DSP in a research paper, please cite our work as follows:
@article{khattab2023dspy,
title={DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines},
author={Khattab, Omar and Singhvi, Arnav and Maheshwari, Paridhi and Zhang, Zhiyuan and Santhanam, Keshav and Vardhamanan, Sri and Haq, Saiful and Sharma, Ashutosh and Joshi, Thomas T. and Moazam, Hanna and Miller, Heather and Zaharia, Matei and Potts, Christopher},
journal={arXiv preprint arXiv:2310.03714},
year={2023}
}
@article{khattab2022demonstrate,
title={Demonstrate-Search-Predict: Composing Retrieval and Language Models for Knowledge-Intensive {NLP}},
author={Khattab, Omar and Santhanam, Keshav and Li, Xiang Lisa and Hall, David and Liang, Percy and Potts, Christopher and Zaharia, Matei},
journal={arXiv preprint arXiv:2212.14024},
year={2022}
}
You can also read more about the evolution of the framework from Demonstrate-Search-Predict to DSPy:
- DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines (Academic Paper, Dec 2023)
- DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines (Academic Paper, Oct 2023)
- Releasing DSPy, the latest iteration of the framework (Twitter Thread, Aug 2023)
- Releasing the DSP Compiler (v0.1) (Twitter Thread, Feb 2023)
- Introducing DSP (Twitter Thread, Jan 2023)
- Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP (Academic Paper, Dec 2022)
Note: If you're looking for Demonstrate-Search-Predict (DSP), which is the previous version of DSPy, you can find it on the v1 branch of this repo.