A CLI tool for processing STDIN or a file with the tiktoken tokenizer.
Options:
--help Show this message and exit.
Commands:
count Prints the number of tokens in the input.
dump Dumps the tokens and their text representations.
head Prints the first N tokens from the input.
split Splits the input text into multiple files.
tail Prints the last N tokens from the input.
pip install git+https://github.com/urbandictionary/token-tail