Introduction | Quick Install | Usage | Documentation | Examples | Contributing | modelfusion.dev
Note
ModelFusion is in its initial development phase. Until version 1.0 there may be breaking changes, because I am still exploring the API design. Feedback and suggestions are welcome.
ModelFusion is a library for building AI apps, chatbots, and agents. It provides abstractions for AI models, vector indices, and tools.
- Type inference and validation: ModelFusion uses TypeScript and Zod to infer types wherever possible and to validate model responses.
- Flexibility and control: AI application development can be complex and unique to each project. With ModelFusion, you have complete control over the prompts and model settings, and you can access the raw responses from the models quickly to build what you need.
- No chains and predefined prompts: Use the concepts provided by JavaScript (variables, functions, etc.) and explicit prompts to build applications you can easily understand and control. Not hidden prompts and logic.
- Multimodal Support: Beyond just LLMs, ModelFusion encompasses a diverse array of models including text generation, text-to-speech, speech-to-text, and image generation, allowing you to build multifaceted AI applications with ease.
- Integrated support features: Essential features like logging, retries, throttling, tracing, and error handling are built-in, helping you focus more on building your application.
npm install modelfusionOr use a template: ModelFusion terminal app starter
You can provide API keys for the different integrations using environment variables (e.g., OPENAI_API_KEY) or pass them into the model constructors as options.
Generate text using a language model and a prompt. You can stream the text if it is supported by the model. You can use prompt formats to change the prompt format of a model.
const text = await generateText(
new OpenAITextGenerationModel({
model: "gpt-3.5-turbo-instruct",
}),
"Write a short story about a robot learning to love:\n\n"
);Providers: OpenAI, Cohere, Llama.cpp, Hugging Face
const textStream = await streamText(
new OpenAITextGenerationModel({
model: "gpt-3.5-turbo-instruct",
}),
"Write a short story about a robot learning to love:\n\n"
);
for await (const textFragment of textStream) {
process.stdout.write(textFragment);
}Providers: OpenAI, Cohere, Llama.cpp
Prompt format lets you use higher level prompt structures (such as instruction or chat prompts) for different models.
const text = await generateText(
new LlamaCppTextGenerationModel({
contextWindowSize: 4096, // Llama 2 context window size
maxCompletionTokens: 1000,
}).withPromptFormat(mapInstructionPromptToLlama2Format()),
{
system: "You are a story writer.",
instruction: "Write a short story about a robot learning to love.",
}
);const textStream = await streamText(
new OpenAIChatModel({
model: "gpt-3.5-turbo",
}).withPromptFormat(mapChatPromptToOpenAIChatFormat()),
[
{ system: "You are a celebrated poet." },
{ user: "Write a short story about a robot learning to love." },
{ ai: "Once upon a time, there was a robot who learned to love." },
{ user: "That's a great start!" },
]
);| Prompt Format | Instruction Prompt | Chat Prompt |
|---|---|---|
| OpenAI Chat | ✅ | ✅ |
| Llama 2 | ✅ | ✅ |
| Alpaca | ✅ | ❌ |
| Vicuna | ❌ | ✅ |
| Generic Text | ✅ | ✅ |
ModelFusion model functions return rich results that include the original response and metadata when you call .asFullResponse() before resolving the promise.
// access the full response and the metadata:
// the response type is specific to the model that's being used
const { output, response, metadata } = await generateText(
new OpenAITextGenerationModel({
model: "gpt-3.5-turbo-instruct",
maxCompletionTokens: 1000,
n: 2, // generate 2 completions
}),
"Write a short story about a robot learning to love:\n\n"
).asFullResponse();
for (const choice of response.choices) {
console.log(choice.text);
}
console.log(`Duration: ${metadata.durationInMs}ms`);Generate a structure that matches a schema.
const sentiment = await generateStructure(
new OpenAIChatModel({
model: "gpt-3.5-turbo",
temperature: 0,
maxCompletionTokens: 50,
}),
new ZodStructureDefinition({
name: "sentiment",
description: "Write the sentiment analysis",
schema: z.object({
sentiment: z
.enum(["positive", "neutral", "negative"])
.describe("Sentiment."),
}),
}),
[
OpenAIChatMessage.system(
"You are a sentiment evaluator. " +
"Analyze the sentiment of the following product review:"
),
OpenAIChatMessage.user(
"After I opened the package, I was met by a very unpleasant smell " +
"that did not disappear even after washing. Never again!"
),
]
);Providers: OpenAI
Generate a structure (or text as a fallback) using a prompt and multiple schemas. It either matches one of the schemas or is text reponse.
const { structure, value, text } = await generateStructureOrText(
new OpenAIChatModel({ model: "gpt-3.5-turbo", maxCompletionTokens: 1000 }),
[
new ZodStructureDefinition({
name: "getCurrentWeather" as const, // mark 'as const' for type inference
description: "Get the current weather in a given location",
schema: z.object({
location: z
.string()
.describe("The city and state, e.g. San Francisco, CA"),
unit: z.enum(["celsius", "fahrenheit"]).optional(),
}),
}),
new ZodStructureDefinition({
name: "getContactInformation" as const,
description: "Get the contact information for a given person",
schema: z.object({
name: z.string().describe("The name of the person"),
}),
}),
],
[OpenAIChatMessage.user(query)]
);Providers: OpenAI
Tools are functions that can be executed by an AI model. They are useful for building chatbots and agents.
Predefined tools: SerpAPI, Google Custom Search
A tool is a function with a name, a description, and a schema for the input parameters.
const calculator = new Tool({
name: "calculator",
description: "Execute a calculation",
inputSchema: new ZodSchema(
z.object({
a: z.number().describe("The first number."),
b: z.number().describe("The second number."),
operator: z
.enum(["+", "-", "*", "/"])
.describe("The operator (+, -, *, /)."),
})
),
execute: async ({ a, b, operator }) => {
switch (operator) {
case "+":
return a + b;
case "-":
return a - b;
case "*":
return a * b;
case "/":
return a / b;
default:
throw new Error(`Unknown operator: ${operator}`);
}
},
});The model determines the parameters for the tool from the prompt and then executes it.
const { tool, parameters, result } = await useTool(
new OpenAIChatModel({ model: "gpt-3.5-turbo" }),
calculator,
[OpenAIChatMessage.user("What's fourteen times twelve?")]
);The model determines which tool to use and its parameters from the prompt and then executes it. Text is generated as a fallback.
const { tool, parameters, result, text } = await useToolOrGenerateText(
new OpenAIChatModel({ model: "gpt-3.5-turbo" }),
[calculator /* and other tools... */],
[OpenAIChatMessage.user("What's fourteen times twelve?")]
);Turn speech (audio) into text.
const transcription = await transcribe(
new OpenAITranscriptionModel({ model: "whisper-1" }),
{
type: "mp3",
data: await fs.promises.readFile("data/test.mp3"),
}
);Providers: OpenAI (Whisper)
Turn text into speech (audio).
// `speech` is a Buffer with MP3 audio data
const speech = await synthesizeSpeech(
new LmntSpeechSynthesisModel({
voice: "034b632b-df71-46c8-b440-86a42ffc3cf3", // Henry
}),
"Good evening, ladies and gentlemen! Exciting news on the airwaves tonight " +
"as The Rolling Stones unveil 'Hackney Diamonds,' their first collection of " +
"fresh tunes in nearly twenty years, featuring the illustrious Lady Gaga, the " +
"magical Stevie Wonder, and the final beats from the late Charlie Watts."
);Providers: Eleven Labs, LMNT
Describe an image as text, e.g. for image captioning or OCR.
const text = await describeImage(
new HuggingFaceImageDescriptionModel({
model: "nlpconnect/vit-gpt2-image-captioning",
}),
data: buffer
);Providers: HuggingFace
Generate a base64-encoded image from a prompt.
const image = await generateImage(
new OpenAIImageGenerationModel({ size: "512x512" }),
"the wicked witch of the west in the style of early 19th century painting"
);Providers: OpenAI (Dall·E), Stability AI, Automatic1111
Create embeddings for text. Embeddings are vectors that represent the meaning of the text.
const embeddings = await embedTexts(
new OpenAITextEmbeddingModel({ model: "text-embedding-ada-002" }),
[
"At first, Nox didn't know what to do with the pup.",
"He keenly observed and absorbed everything around him, from the birds in the sky to the trees in the forest.",
]
);Providers: OpenAI, Cohere, Llama.cpp, Hugging Face
Split text into tokens and reconstruct the text from tokens.
const tokenizer = new TikTokenTokenizer({ model: "gpt-4" });
const text = "At first, Nox didn't know what to do with the pup.";
const tokenCount = await countTokens(tokenizer, text);
const tokens = await tokenizer.tokenize(text);
const tokensAndTokenTexts = await tokenizer.tokenizeWithTexts(text);
const reconstructedText = await tokenizer.detokenize(tokens);Providers: OpenAI, Cohere, Llama.cpp
const texts = [
"A rainbow is an optical phenomenon that can occur under certain meteorological conditions.",
"It is caused by refraction, internal reflection and dispersion of light in water droplets resulting in a continuous spectrum of light appearing in the sky.",
// ...
];
const vectorIndex = new MemoryVectorIndex<string>();
const embeddingModel = new OpenAITextEmbeddingModel({
model: "text-embedding-ada-002",
});
// update an index - usually done as part of an ingestion process:
await upsertIntoVectorIndex({
vectorIndex,
embeddingModel,
objects: texts,
getValueToEmbed: (text) => text,
});
// retrieve text chunks from the vector index - usually done at query time:
const retrievedTexts = await retrieve(
new VectorIndexRetriever({
vectorIndex,
embeddingModel,
maxResults: 3,
similarityThreshold: 0.8,
}),
"rainbow and water droplets"
);Available Vector Stores: Memory, Pinecone
Integrations: Helicone
Examples for almost all of the individual functions and objects. Highly recommended to get started.
Terminal app, chat, llama.cpp
Next.js app, OpenAI GPT-3.5-turbo, streaming, abort handling
A web chat with an AI assistant, implemented as a Next.js app.
terminal app, PDF parsing, in memory vector indices, retrieval augmented generation, hypothetical document embedding
Ask questions about a PDF document and get answers from the document.
Next.js app, Stability AI image generation
Create an 19th century painting image for your input.
Next.js app, OpenAI Whisper
Record audio with push-to-talk and transcribe it using Whisper, implemented as a Next.js app. The app shows a list of the transcriptions.
terminal app, agent, BabyAGI
TypeScript implementation of the BabyAGI classic and BabyBeeAGI.
terminal app, ReAct agent, GPT-4, OpenAI functions, tools
Get answers to questions from Wikipedia, e.g. "Who was born first, Einstein or Picasso?"
terminal app, agent, tools, GPT-4
Small agent that solves middle school math problems. It uses a calculator tool to solve the problems.
terminal app, PDF parsing, recursive information extraction, in memory vector index, _style example retrieval, OpenAI GPT-4, cost calculation
Extracts information about a topic from a PDF and writes a tweet in your own style about it.
Read the ModelFusion contributing guide to learn about the development process, how to propose bugfixes and improvements, and how to build and test your changes.