OpenPipe is a flexible playground for comparing and optimizing LLM prompts. It lets you quickly generate, test and compare candidate prompts, and can automatically translate those prompts between models.
You can use our hosted version of OpenPipe at https://openpipe.ai. You can also clone this repository and run it locally.
These are simple experiments users have created that show how OpenPipe works. Feel free to fork them and start experimenting yourself.
- All models available through the OpenAI chat completion API
- Llama2 7b chat, 13b chat, 70b chat.
- Anthropic's Claude 1 Instant and Claude 2
Inspect prompt completions side-by-side.
OpenPipe lets you template a prompt. Use the templating feature to run the prompts you're testing against many potential inputs for broad coverage of your problem space.
Write your prompt in one format and automatically convert it to work with any other model.
![Screenshot 2023-08-01 at 11 55 38 PM](https://private-user-images.githubusercontent.com/41524992/257726994-1e19ccf2-96b6-4e93-a3a5-1449710d1b5b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjUwOTYzNDcsIm5iZiI6MTcyNTA5NjA0NywicGF0aCI6Ii80MTUyNDk5Mi8yNTc3MjY5OTQtMWUxOWNjZjItOTZiNi00ZTkzLWEzYTUtMTQ0OTcxMGQxYjViLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MzElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODMxVDA5MjA0N1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWZjYzdlYmEwYThkZWI1MjM2YjA4ZmU4ZTQwN2VhNTBhZmQzMDg4NDBiN2IwM2Q2YWI5OGNiZjI5ZjkzZDRlMTYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.DsqLSBYGXLhmOLypKpzdEUGBngAKZI9pKqrdUlw_02Y)
Use a growing database of best-practice refinements to improve your prompts automatically.
![Screenshot 2023-08-01 at 11 55 38 PM](https://private-user-images.githubusercontent.com/41524992/257734697-87a27fe7-daef-445c-a5e2-1c82b23f9f99.gif?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjUwOTYzNDcsIm5iZiI6MTcyNTA5NjA0NywicGF0aCI6Ii80MTUyNDk5Mi8yNTc3MzQ2OTctODdhMjdmZTctZGFlZi00NDVjLWE1ZTItMWM4MmIyM2Y5Zjk5LmdpZj9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MzElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODMxVDA5MjA0N1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTk5YTM1ZTc1YzE2ZDk3ZGNmYTVkN2ZjYTBmYTY5Mjk4NWJmYjAwMzBlZDNmNWY4YWNkYjc2ODdkMmEwN2YyYzgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.kJioLtaRh5VHgyu93eWuYN-9TbTnRNjoR_i8JMLl0T4)
OpenPipe includes a tool to generate new test scenarios based on your existing prompts and scenarios. Just click "Autogenerate Scenario" to try it out!
- Install Postgresql.
- Install NodeJS 20 (earlier versions will very likely work but aren't tested).
- Install
pnpm
:npm i -g pnpm
- Clone this repository:
git clone https://github.com/openpipe/openpipe
- Install the dependencies:
cd openpipe && pnpm install
- Create a
.env
file (cp .env.example .env
) and enter yourOPENAI_API_KEY
. - Update
DATABASE_URL
if necessary to point to your Postgres instance and runpnpm prisma db push
to create the database. - Create a GitHub OAuth App and update the
GITHUB_CLIENT_ID
andGITHUB_CLIENT_SECRET
values. (Note: a PR to make auth optional when running locally would be a great contribution!) - Start the app:
pnpm dev
. - Navigate to http://localhost:3000