isWithinTokenLimit fails due to lack of model being provided
Closed this issue · 7 comments
Unsure if this is just a documentation issue, however, after checking the source code there appears to be no default or clear instruction on how to provide the model type for tokenization in the isWithinTokenLimit function.
I'll happily open a PR on this but want to know if there's any kind of contribution guidelines.
To reproduce, run as such isWithinTokenLimit(messages, MAX_CHAT_LENGTH) where messages is a ChatMessage iterable and MAX_CHAT_LENGTH is a token length the chain should not be longer than.
I am having the same issue, I get an error when trying to use isWithinTokenLimit without passing a model_name. even more when I try to use just encodeChat passing the messages and model, I get the following error:
TypeError: Cannot read properties of undefined (reading 'get')
at Object.encodeChatGenerator (/workspace/node_modules/gpt-tokenizer/cjs/GptEncoding.js:99:57)
at encodeChatGenerator.next ()
at Object.encodeChat (/workspace/node_modules/gpt-tokenizer/cjs/GptEncoding.js:141:25)
Joining the issue
just FYI guys, as an alternative to this package I have been using : https://www.npmjs.com/package/gpt-tokens; https://github.com/Cainier/gpt-tokens
is working fine and offers similar functionalities
just FYI guys, as an alternative to this package I have been using : https://www.npmjs.com/package/gpt-tokens; https://github.com/Cainier/gpt-tokens
is working fine and offers similar functionalities
Works well, thanks!
I had this issue, I fixed it by adding the model to the import:
const { isWithinTokenLimit } = require('gpt-tokenizer/model/gpt-4-0314');
Hope this helps
In order to select the model for isWithinTokenLimit, you need to import the file that represents correct model / encoding.
🎉 This issue has been resolved in version 2.2.1 🎉
The release is available on:
Your semantic-release bot 📦🚀