/tokenizer-app

Like tiktokenizer on the web - but for all huggingface models

Primary LanguageJavaScript

Tokenization

This is a replica of Tiktokenizer, but for all models on HF hub

Instructions

Server

poetry install - to setup the python environment

Set the HUGGINGFACE_API_TOKEN environment variable

Run server.py

UI

yarn start