A simple script to explore the token generation process of an LLM. Insert a user prompt and explore the tokens generated by an LLM. It defaults to HuggingFaceTB/SmolLM2-135M-Instruct but you can change it in the script by changing the model_id variable.
If you do not have uv installed, follow the uv installation instructions to install it. Then run:
uv run https://raw.githubusercontent.com/alonsosilvaallende/nextToken/refs/heads/main/nextToken.pyFor a more complex library but with more features, check the token-explorer project.