Use custom tokenizer
Opened this issue · 0 comments
linonetwo commented
To fix #104 , I think I should use https://github.com/belladoreai/llama-tokenizer-js , but currently there is no such an API to passing vector to createCompletion
Opened this issue · 0 comments
To fix #104 , I think I should use https://github.com/belladoreai/llama-tokenizer-js , but currently there is no such an API to passing vector to createCompletion