Redis, known as a data structure server, is often used to store and retrieve distributed data, mostly for caching or distributing messages. Yet solutions using redis have some limitations:
- Client code has to use explicit commands (HGET, SMEMBERS, etc) to store and retrieve objects, instead of native data structure operators and methods that people are familiar with.
- Redis can store only one level of data structures - a list, a dict, or a set of strings. Nested structures like a list of dicts, a dict with list values, are not natively supported.
Dmem is a thin wrapper over Redis, that provides a transparent API in python to store/retrieve data on remote servers:
- It uses redis API behind the scenes, but provides data operations via well-known builtin type operators and methods.
- Nested structures of multiple levels are supported.
- A simple reference counting mechanism is in place, so when a remote object is no used by any client code, it is deleted from Redis.
- A distributed lock is in place, to protect a dmem object from multiple access from different clients
Dmem provides 5 new types that maps python container types to Redis data structures: RedisStr, RedisList, RedisDict, RedisSet, RedisObject, you can use them exactly like the native counterparts(str/list/dict/set/object), except of course, the data is stored in another Redis node.
Before instantiating any dmem objects, set up your Redis nodes configuration:
from dmem import *
# Configure your redis instances pool
RedisClientPool.get_pool().load_config({
"redis1": {"host":"192.168.1.1", "port": 6379, "db":0},
"redis2": {"host":"192.168.1.2", "port": 6379, "db":0},
"redis3": {"host":"192.168.1.3", "port": 6379, "db":0},
})
Now it is time to play:
mylist = RedisList([1, "abc", 3.1415])
# Now the list is alive on Redis
print mylist[1] # actually retrieved with LGET command
# output: 'abc'
print mylist[0:2] # LRANGE command
# output: [1, 'abc']
del mylist[0] # LREM command
print mylist[:]
# output: ["abc", 3.1415]
s = RedisStr("Redis string")
# Now the string lives in Redis
mylist.append(s) # APPEND command
print len(mylist) # STRLEN command
# output: 3
print mylist[-1].getvalue() # GET command
# output: 'Redis string'
mydict = RedisDict({"a":1234}) # HSET commands
# Now the dict lives in Redis as a hashmap
mydict["list"] = mylist # HGET command
print mydict.keys() # HKEYS command
# output: ['a', 'list']
for k,v in mydict.items(): # HGETALL command
print k, v
# output: a 1234
# output: list <redislist.RedisList object at 0x01CDEF30>
print mydict["list"][0]
# output: 'abc'
obj = RedisObject()
# For redis object, the attributes are stored in Redis
obj.attr1 = "abc"
obj.attr2 = mydict
print obj.attr2['list'][0]
# output: 'abc'
As you can see, you can use dmem objects almost anywhere a native python type is used. And for applications that need a lot of main memory, you don't need to worry about OOM problems, as long as you have enough Redis nodes configured.
The only downside to using dmem objects is: data access is less efficient than main memory, as a network roundtrip is involved. Sometimes, multiple roundtrips can be saved, if you preload a dmem object in one go. For instance, when iterating over a RedisList:
# without preloading
for v in mylist:
print v # Every item access is a LGET command
# with preloading
with mylist.loaded() as cache:
for v in cache:
print v
Dmem is fairly straightforward:
- It gives each dmem object a unique address, and use it as KEY in redis.
- It maps data structure operators/methods to Redis commands, pipeline/lua scripts are used to pack multiple command into one network request.
- Both value and data type are preserved, if you save
1.23in a dmem container, you will get afloatback, instead of astring'1.23'
If you want to see what exactly is happening, just turn on debug, and see all Redis commands printed:
>>> dmem.enable_debug()
>>> l = dmem.RedisList([1,2,3])
Calling command getset of redis client
Calling command rpush of redis client
Calling command rpush of redis client
Calling command rpush of redis client