Calculated tokens much higher than actual

Question

Calculated tokens much higher than actual

Closed this issue a year ago · 10 comments

Thanks for this. I've noticed a weird issue though both with this library and also the official code from open ai that I found a while back before gpt-4 came out.

What is happening is that the tokens calculated by this tool are much higher than the openai api is reporting in the completion. For example, a prompt I just submitted to gpt-4 was calculated as 7810 tokens by this library but when I got the completion from openai it told me my prompt had 5423 tokens. I'm not sure if you have also noticed something similar? In the prompt I'm submitting primarily Node JS code.

Answer 1 · 2023-05-14T15:48:52.000Z

As a workaround, I've noticed when you request too many tokens, you get a 400 error very quickly for example

This model's maximum context length is 8192 tokens. However, you requested 13674 tokens (7469 in the messages, 6205 in the completion). Please reduce the length of the messages or completion.

So I parse of the messages token count and resubmit with a max_tokens calculated as follows: 8192 - 7469 - 1

Answer 2 · 2023-05-21T02:17:43.000Z

Hi @Qarj! Thanks for flagging this problem.
As @ricardomatias noticed in #5, the tokenizer is using the r50k_base encoding, which isn't the one used by GPT-4. Hence the token offset. I'm working on v2 which will allow for choosing which encoding to use, which will correctly tokenize for GPT-4 specifically.

Answer 3 · 2023-05-22T09:47:26.000Z

Thanks very much for addressing this! I will definitely use this feature in v2 when it is out.

Answer 4 · 2023-05-23T16:59:16.000Z

🎉 This issue has been resolved in version 2.0.0-beta.1 🎉

The release is available on:

Your semantic-release bot 📦🚀

Answer 5 · 2023-05-23T22:27:59.000Z

Thanks very much for this! Am using it already :)

Answer 6 · 2023-05-23T23:11:58.000Z

So it seems it is much closer now to the actual tokens, in a test I did the prompt was calculated as 998 tokens according to the library but 1003 tokens according to open ai. I suspect if we allow a 50 token margin then our completion token requests should always be within limit.

Answer 7 · 2023-05-24T01:54:53.000Z

Interesting. I wonder if there are 5 additional tokens that are set by OpenAI for each request? The algorithm should be exactly the same as OpenAIs.

Thanks for investigating.

Answer 8 · 2023-05-24T02:03:54.000Z

🎉 This issue has been resolved in version 2.0.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Answer 9 · 2023-06-03T01:27:28.000Z

@Qarj added the new encodeChat function which should return correct values for chats!

Answer 10 · 2023-06-03T22:55:54.000Z

Thanks very much for this! :)