A ChatGPT tokenizer implementation for .NET. You can use this library to estimate the cost (amount of tokens) of your request to ChatGPT.
The library is available on NuGet. Just search for ChatGptNet.Tokenizer in the Package Manager GUI or run the following command in the .NET CLI:
dotnet add package ChatGptNet.Tokenizer
This is a C# implementation of OpenAI's original python encoder/decoder which can be found here. This implementation was strongly inspired from the JavaScript version created by latitudegames, that can be found here.
The library can be used in any .NET application built with .NET 6.0 or later. Just create an instance of the GptTokenizer and you are ready to go.
You can either create the default tokenizer
var tokenizer = await GptTokenizer.CreateTokenizerAsync();Or, optionally, you can create a tokenizer with a custom vocab.bpe and encodings.json source files with:
using var vocabStream = File.OpenRead(@"path/to/vocab.bpe");
using var encodingStream = File.OpenRead(@"path/to/encodings.json");
var tokenizer = await GptTokenizer.CreateTokenizerAsync(vocabStream, encodingsStream);The default vocab.bpe and encodings.json are already included in this library.
Once you obtain an instance of the GptTokenizer, you can perform the tokenization of your text.
var tokens = tokenizer.GetTokens(@"Lorem ipsum dolor sit amet")
var tokensCount = tokens.Length;
var tokensAsText = tokenizer.GetTextFromTokens(tokens);This package extends the ChatGptNet NuGet package (source: https://github.com/marcominerva/ChatGptNet) by providing some functionalities for estimating the token amount.
public async Task Estimate(IChatGptClient chatGptClient)
{
var conversationId = Guid.NewGuid();
chatGptClient.EstimateConversationTokenCountAsync("first message");
chatGptClient.EstimateConversationTokenCountAsync(conversationId);
chatGptClient.EstimateConversationTokenCountAsync(conversationId, "next message");
}
Contributions are welcome. Feel free to file issues and pull requests on the repo and we'll address them as we can.