JerryFans/flutter_tiktoken

flutter_tiktoken is a flutter offline package for a fast BPE tokeniser for OpenAI models.

flutter_tiktoken

flutter_tiktoken is a flutter offline package for a fast BPE tokeniser for OpenAI models.

flutter_tiktoken is a copy package in https://github.com/mvitlov/tiktoken and fixed the issue build spent many time (more than one hour) when build in release mode.

Open Ai Official Calcuate	flutter_tiktoken

Usage

init flutter_tiktoken

void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await TiktokenDataProcessCenter().initData();
  runApp(const MyApp());
}

calcuateTokens

static int _calcuateTokens(
      {required String text, String modelName = 'gpt-4'}) {
    final encoding = encodingForModel(modelName);
    final numTokens = encoding.encode(text).length;
    return numTokens;
  }