impl of computing a message's token count for providers other than OpenAI
Intex32 opened this issue · 0 comments
Intex32 commented
In LLM there is a function called tokensFromMessages. Current default implementation is using the models encoding (from ModelType) to compute the token count locally.
Problem: Afaik, the encoding is not made publicly available by Google. Thus we have to make an API call to GCP (https://cloud.google.com/vertex-ai/docs/generative-ai/get-token-count).
TODO: default implementation of tokensFromMessages has to be removed and replaced by provider specific implementations (for OpenAI based on encoding, and for GCP on external API call)