effectai/effect-js

Implement IPFS cache system

Closed this issue · 2 comments

IPFS objects should be cached to avoid too many HTTP calls.

This should be done by a generic cache interface. The default cache implementation should be an in-memory one (perhaps a Map). On initializing the SDK users should be able to provide an alternative cache implementation for persistence. For web applications this will likely be LocalStorage, which should be provided by the SDK as an import. React Native and NodeJS apps will be able to provide their own implementations.


Side notes
The complexity of this cache system is still up for discussion. A more advanced system could implement a key expiry setting, to allow the cache to be used for get_table_rows. And it could perhaps item loading states to avoid asynchronous loading the same item into the cache. These advanced features may be explored but are not mandatory right now.

In #105 a simple caching system is implemented inside the IPFS service's fetch method, which defaults to the default options configured in the client (10 minutes):

fetch = async (
    hash: string,
    ipfsContentForm: IpfsContentFormat = IpfsContentFormat.JSON,
    cacheTimeInMs = this.client.options.ipfsCacheDurationInMs,
 )

if the cache time is truthy we use it to cache the ipfs responses for the duration.
if falsy we dont cache.

    if (cacheTimeInMs) {
        // Create a cache key
        const cacheKey = `${hash}-${ipfsContentForm}`;

        // If we have the response cached, return it
        const cachedItem = await get(cacheKey);
        if (cachedItem && Date.now() < cachedItem.timestamp + cacheTimeInMs) {
          return cachedItem.data;
        }
      }

      .... ifpfs data logic ....

      // After we got the result, cache it
    if (cacheTimeInMs) {
        await set(cacheKey, { data: result, timestamp: Date.now() });
      }

we can then opt-out of caching when initializing the client

  const client = shallowRef(
    new Client(jungle4, {
      ipfsCacheDurationInMs: null, <-- can also be 0 to turn off caching.
    }),
  );

I was looking into idb-keyval and a concern I have is that it's coupled to IndexedDB (a browser API) and might lack support for other runtimes (like React Native, NodeJS)

Given that it's quite a small library perhaps we can intern it and make it more generic:

  1. Make an abstract interface for get/set , and add a default in-memory cache implementation
  2. Add an additional IndexedDB implementation for browsers