Firebase Genkit <> Promptfoo Plugin

Promptfoo Community Plugin for Google Firebase Genkit

genkitx-promptfoo is a community plugin for using Promptfoo with Firebase Genkit. Built by Yuki Nagae.

This Genkit plugin allows to use Promptfoo.

Warning

This version is experimental and may have API changes and critical bugs. Use it for prototypes or hobby projects, not in production.

Installation

Install the plugin in your project with your favorite package manager:

npm install genkitx-promptfoo
yarn add genkitx-promptfoo
pnpm add genkitx-promptfoo

Usage

Configuration

import { promptfooEval } from 'genkitx-promptfoo';

configureGenkit({
  plugins: [
    promptfooEval({
      metrics: [
        {
          type: 'contains',
          value: 'Hello, World!',
        },
        {
          type: 'regex',
          value: '^Hello, World!$',
        },
        {
          type: 'javascript',
          value: "output.includes('Hello, World!')",
        },
        {
          type: 'similar',
          value: 'Aloha, World!',
          threshold: 0.8,
          provider: 'vertex:embedding:text-embedding-004',
        },
        {
          type: 'llm-rubric',
          value: 'It is a friendly greeting.',
          provider: 'vertex:gemini-1.5-flash',
        },
      ],
    }),
  ],
});

Basic examples

You can specify the Promptfoo metrics below:

contains / icontains

{
  type: 'contains',
  value: 'The expected substring',
},

contains-all

{
  type: 'contains-all',
  value: [
   'Value 1',
   'Value 2',
   'Value 3',
  ],
},

contains-any

{
  type: 'contains-any',
  value: [
   'Value 1',
   'Value 2',
   'Value 3',
  ],
},

regex

{
  type: 'regex',
  value: '\\d{4}', // Matches a 4-digit number
},

contains-json / is-json

{
  type: 'contains-json',
  value:
  {
    'required': ['latitude', 'longitude'],
    'type': 'object',
    'properties':
      {
        'latitude': { 'type': 'number', 'minimum': -90, 'maximum': 90 },
        'longitude': { 'type': 'number', 'minimum': -180, 'maximum': 180 },
      },
  }
},

contains-sql / is-sql

To use this assertion, you need to install the node-sql-parser package. You can install it using npm: npm install node-sql-parser.

{
  type: 'contains-sql',
},

equals

{
  type: 'equals',
  value: 'The expected output',
},

contains-xml / is-xml

{
  type: 'contains-xml',
},

javascript

{
  type: 'javascript',
  value: "output.includes('Hello, World!')",
},

starts-with

{
  type: 'starts-with',
  value: 'Yes',
},

similar

{
  type: 'similar',
  value: 'The expected output',
  threshold: 0.8,
  provider: 'vertex:embedding:text-embedding-004',
},

llm-rubric

{
  type: 'llm-rubric',
  value: 'Is not apologetic and provides a clear, concise answer',
  provider: 'vertex:gemini-1.5-flash',
},

Contributing

Want to contribute to the project? That's awesome! Head over to our Contribution Guidelines.

Need support?

Note

This repository depends on Google's Firebase Genkit. For issues and questions related to Genkit, please refer to instructions available in Genkit's repository.

Reach out by opening a discussion on Github Discussions.

Credits

This plugin is proudly maintained by Yuki Nagae Yuki Nagae.

I got the inspiration, structure and patterns to create this plugin from the Genkit Community Plugins repository built by the Fire Compnay as well as the github plugin.

License

This project is licensed under the Apache 2.0 License.

yukinagae/genkitx-promptfoo