AbanteAI/spice

Anthropic token counting

PCSwingle opened this issue · 1 comments

Anthropic doesn't have any official way to count tokens; the best way is probably to just estimate token counts. Maybe we want to add a token estimate boolean to SpiceResponse to indicate that it's an estimate, and we would set it to true whenever we didn't get the tokens directly from the api (even for openai which we have hopefully accurate token counting functions)?

I think we can be certain of the OpenAI counts since tiktoken is their library. So instead of adding an estimate bool we could just say that token counts may be approximate for a SpiceResponse that was got through current_response, but should be accurate for any response from complete_response