Allow the injection of TokenCountEstimator

Question

Allow the injection of TokenCountEstimator

Closed this issue 23 days ago · 5 comments

andreadimaio commented a month ago

Today it is not possible to inject the TokenCountEstimator, this could be useful to estimate the number of tokens.

geoand commented 25 days ago

Thanks!

Answer 1 · 2024-06-25T07:10:25.000Z

Where would one inject that?

Answer 2 · 2024-06-25T07:23:42.000Z

The idea is to have the same configuration of the ChatLanguageModel, in this case the developers can inject the interface where they prefer.

For example, suppose I have two AIServices configured with watsonx, one with llama3 and the other with mixstral. The developer could inject the TokenCountEstimator for both models and choose which to use based on the token counts.

This is an idea of what I have in mind LINK.

Answer 3 · 2024-06-25T07:41:16.000Z

We can certainly do that and it does makes sense!

Answer 4 · 2024-06-25T08:22:12.000Z

Ok, I'm going to share what I wrote (I have more tests to add) for the end of the day.