How to use disk to store data
Closed this issue · 1 comments
Does pysyncobj support storing data in disk? in the case that data is too big to store in RAM.
I modified the examples code, and use a bsddb object to store data, instead of self.__data
, which is a dict.
I works well, when I just read and insert some key-values. But, I am afraid that it can not replicate snapshot if I add a new node. What is the best-practice to implement it by using PySyncObj?
You need to write custom serializer, deserializer and (if you need async serialization) - serializeChecker. You can, for example, create tar.gz with your bsddb disk based file. You can look at https://github.com/bakwc/DbSyncer, eg. sqlite_driver.py
Async serialization could be a tricky one so I think sync one is ok. There is an option that separates full serialization at time on different nodes so that when one node performs a sync serialization - other nodes still able to perform responses.