Tokenization In this repository, you can find a set of tokenizers and tests for them. The tokenizers segment a given text into tokens of different types, e.g., alphabetic, numeric, alphanumeric sequences, punctuation.