/tiktokenizer-js

A static site for visualizing the GPT-2 Byte-Pair Encoding (BPE) tokenization, replicating the official OpenAI API

Primary LanguageJavaScript

Stargazers