seanzhang-zhichen/expand-baichuan-tokenizer

扩充百川大模型词表，其他模型也类似

Python

RUN

pip install -r requirements.txt
cd src
python main.py