How to calculate usage?
dmki opened this issue · 2 comments
Hi,
In GPT SDKs we have 2 properties of response - Input tokens and output tokens that were used. If I understand right, here we are getting only strings of Claude's response, so how would you suggest to calculate the cost of the call?
Thanks!
This actually sent me down quite the rabbit hole. They haven't really documented how their tokens are calculated in depth, and nothing is returned from the API, you have to dive into their typescript/python sdks, where they are using Huggingface's rust/python tokenizer library with their own custom BPE model (aka different from OpenAI's models). The short version is, I was able to extract from that model, the merges and the vocab files, and use ML.NET to import them and calculate the tokens used in a string. It'll add a few dependencies, which I need to sort through, but TLDR, I think I can work out an addition that lets you accurately calculate tokens. I'll get back to you in a couple of days when I've got something stable and tested.
Version 1.2 has been released, which includes an extension method for counting tokens accurately. Please see the readme or the tests for example usages! Thanks for raising the issue.