Feature: Tokenizer
Opened this issue · 0 comments
billabongrob commented
It would be really cool to have a Tokenizer similar to https://www.npmjs.com/package/tiktoken available to approximate token counts on text in order to estimate costs in certain tools.
The nice part about this would be that it keeps the data internal to your organization so you don't have to post into external tools.