Use weak references for internal caches

[hurricup]

Initial idea was to support refcount and use weak references for internal caches.
But for now it's just normal one. May hit at some point.
We need to understand the lifetime of the cache and weaken refs if necessary.