/dn8-sk-llama3-fireworks

A repository demonstrating .NET 8 using Semantic Kernel to generate multiple streams of content using System.Threading.Channels with Llama 3 and Fireworks.ai

Primary LanguageC#

Need for Speed: LLMs Beyond OpenAI with .NET 8 SSE + Channels, Llama3, and Fireworks.ai

This repository demonstrates how to combine:

  • .NET 8 Web APIs
  • .NET 8 channels
  • Microsoft Semantic Kernel
  • Server Sent Events (SSE)
  • Meta Llama 3 70B on Fireworks.ai

To build a high throughput recipe generator.

To see the full writeup, check out the article: Need for Speed: LLMs Beyond OpenAI with .NET 8 SSE + Channels, Llama3, and Fireworks.ai

Running the Demo

Follow these instructions to run the demo:

First, get an API key from Fireworks.ai and Groq.com and add them to the user secrets.

cd api
dotnet user-secrets init
dotnet user-secrets set "RecipesConfig:FireworksKey" "YOUR_KEY_HERE"
dotnet user-secrets set "RecipesConfig:GroqKey" "YOUR_KEY_HERE"

To run the front-end:

cd web
yarn        # Restore packages
yarn dev    # Run the front-end demo.

In another terminal session:

cd api
dotnet run