daquexian/faster-rwkv

How can i change default vocabs for world model?

xiaol opened this issue a year ago · 1 comments

xiaol commented a year ago

need some change for default vocabs file

like this one, add some new special tokens in the end of file:

https://huggingface.co/xiaol/RWKV-paper-reviewer-and-reading-comprehension-wenda-Worldv4-7B-16k/blob/main/rwkv_vocab_v20230424.txt

daquexian commented a year ago

try https://github.com/daquexian/faster-rwkv/blob/master/tools/convert_tokenizer.py to generate the faster-rwkv tokenizer model from vocab file.