inrupt/pod-server

Redis backend

michielbdejong opened this issue · 8 comments

We want to switch from in-memory storage to, I think, Redis. This creates a number of interesting issues, assuming we want to enable multi-threaded access to Redis.

We could do content-addressable blobs and memberlistings. A memberlisting is a list of members, with for each one:

  • isContainer
  • name
  • etag

Based on name you can get the full path, then add the etag to get the key, and you can then retrieve the resource, which may again be a container member listing or a blob for ResourceData (contentType + body)

To get multi-threading support, there are various options (locking, append-then-check, maybe other options), but I think the easiest one with best result may be to implement conditional updates with Lua scripting.

Maybe one script for create, one for update, one for delete.

eval "if(redis.call('exists', KEYS[1]) == 0) then redis.call('set', KEYS[1], ARGV[1]) return 'yes' else return 'no' end" 1 foo bar
eval "if(redis.call('get', KEYS[1]) == ARGV[1]) then redis.call('set', KEYS[1], ARGV[2]) return 'yes' else return 'no' end" 1 foo bar baz
eval "if(redis.call('get', KEYS[1]) == ARGV[1]) then redis.call('del', KEYS[1]) return 'yes' else return 'no' end" 1 foo baz

Container listings should be hashes so that you can idempotently insert without reading first. But this should still happen in a transaction together with the resource create, because it might race creations with deletions.

Note that creations propagate all the way to the root, but deletions only remove from the directly containing container.

So creating a resource should be executing this first lua script, plus doing a HSET for all the members in containers up to the root.

Updating a resource does not affect the containing containers.

Deleting a non-container resource removes it from its direct container, but the ancestors don't change (there is no support for tree-based sync like in remoteStorage).

Deleting a container resource also deletes all its descendants. I think I can use WATCH for this, see https://redis.io/topics/transactions

WATCH mykey
val = GET mykey
val = val + 1
MULTI
SET mykey $val
EXEC

another option is to say container delete fails if the container is not empty, according to nodeSolidServer/node-solid-server#1179 that's what NSS does.

We want to switch from in-memory storage to, I think, Redis.

Concern: ease of deployment. Redis should likely be one option.

oh yes, the backend is swappable, that goes without saying :)

👍 Excellent. Is there a pointer to commits?