tryAGI/Tiktoken

Questions about cache usage

Closed this issue · 2 comments

JadynWong commented

I used this library to replace "TiktokenSharp," and the library seems to perform better.
However, I noticed that this library implements caching and enables it by default, storing the encoded content or encode count in the FastCacheDictionary.

I'm unsure if there are any issues with my usage. I only use "cl100k_base" and "p50k_base" encodes. So I initialize them at application startup and use them as single instances. Could this potentially lead to memory problems, such as memory overflow?

I am aware that the cache can be disabled, but I was hoping you could provide some suggestions on its best usage.

Thanks.

HavenDV commented

Hello. Thanks for the question.
Didn't quite understand you, you can use Lazy<Tiktoken.Encoding> in a static class if you need a singleton.
If the question is about whether an overflow can occur due to a cache - I highly doubt it. The order of values here is quite small, the whole bpe(100000 values) weighs 1 mb
I checked the absolute values of the used memory and these are:
20 mb is used after getting Encoding and preparing all Dictionaries.
21-22 mb is used after calling Encode for Bitcoin whitepaper.

Undoubtedly, it is more expensive in terms of occupied memory than other available tokenizers. But I very much doubt that it is possible to fill the memory with a cache or even approach some large values ​​like 1 GB

Ideally, we could add a maximum limit on the memory used and a check for that, but I'm not sure if we need to think about that now.

  • 👍1
JadynWong commented

Thanks for the quick reply, my question was indeed about whether the cache would cause a memory overflow.

This is because I can't predict exactly how much memory it will take up as a singleton instance as the user uses it. It may be small at first for a while, but it will continue to grow as it is used. It currently has no limit unless I restart the application or stop referencing the instance. I think it would be worthwhile to implement a maximum limit for the cache.

I tried running benchmarks with the EnableCache to false and this library still performs well. I will use the it as singleton instance by disable cache.

Thanks again!