Simple CLI wrapper for the Open AI tokenizer tiktoken.
Either pipe input into tt.py or pass it a file as the first argument to
tokenize the text (by default using the cl100k_base encoding).
# tokenize the text 'Hello, World!'
echo "Hello, World!" | tt.py > hello-world-tokens.txt
# count the number of tokens
wc -w hello-world-tokens.txt
# decode the tokens
< hello-world-tokens.txt tt.py --decode > hello-world.txtThe main motivation for this script is to be able to quickly count tokens of a given text.
For example, in a Vim buffer
:w !tt.py | wc -wcounts the number of tokens in the current buffer. Which handy if you want to understand if the input is appropriate for a given model/context window.
For example, this README.md contains at the time of writing of 238 tokens. So
it fits into a small context window without any problem.
Requires tiktoken to be installed.
For example using pip install tiktoken.