mlc-ai/tokenizers-cpp

is there any plan to support Tiktoken?

Jasonsey opened this issue · 3 comments

Jasonsey commented

I am trying to make mlc to support Qwen, but the model's tokenizer use tiktoken developed by openai, which is not supported by this repo now? So is there any plan for this feature?

tqchen commented

As of now we don't have planned effort yet, contribution is welcomed

gaokao123 commented

@Jasonsey for tiktoken of qwen, Is there a solution now?

Jasonsey commented

@Jasonsey for tiktoken of qwen, Is there a solution now?

Currently, we can use the mlc chat to convert Qwen'tiktoken to this repo's tokenizer format.