/llm-text-tokenization

Primary LanguageHTMLApache License 2.0Apache-2.0

Visualize Google Cloud Vertex AI large language models and their tokenization

With this application, you can better understand how large language models tokenize your text.

Enter your text, select a model, and see how it is tokenized!

Models currently supported

  • text-embedding
  • text-multilingual-embedding
  • textembedding-gecko
  • textembedding-gecko-multilingual
  • text-bison
  • text-unicorn
  • chat-bison
  • code-gecko
  • code-bison
  • codechat-bison

Try it out!

You can try this application online, and select one of the available PaLM-based models, and see how your text is tokenized.