Question: How frequently can stores be created?
Closed this issue · 3 comments
Can dpsCreateStore be used to create temporary (but persistent) stores frequently, e.g. every 10 seconds, or is it a more heavyweight operation? E.g. create a store related to a cut within a consistent region?
Hi Dan,
It takes around 150 to 300 milliseconds to create a new DPS store in Redis. It is important to know what and how much data will be stored inside of such temporary stores. Current implementation of consistent region done by Fang already supports Redis as a mechanism to store checkpoint data for a given consistent region. It may be necessary to ask a question what will be the major difference in using Redis directly as it is done in the current consistent region implementation versus using such temporary DPS stores.
Such a store would be for application data related supporting exactly once processing with a key-value store with consistent region, so in additon to operator state checkpoints.
Hi Dan,
If sub-second latencies are needed, then it will be a problem via TCP based transport to reach out to the distributed KVS back-end engines such as Redis, Aerospike, Cassandra etc. If RDMA via RoCE can be exploited, IBM's upcoming HSA KVS can be a good choice as it attains maturity in the next year or two,