eclipse-zenoh/roadmap

Storage alignment - Time series support

Mallets opened this issue · 0 comments

Summary

Storages on Zenoh subscribing to the same key expression might diverge due to missing samples. Currently there is no way to converge these storages.
Time series data stores also might have different purging policies. It is different from key-value stores in the amount of metadata required to have a log of all updates, hence need a different strategy.

Intended outcome

The time series datastores even though diverged temporarily, will eventually receive all updates.
All the values until the highest lower bound (considering the oldest values across all datastores) in time-series datastore that subscribe to the same key expression will be the same.

How will it work?

A background protocol periodically generates digests, based on the latest stable state of the time series data store, and publishes it to other time series data stores subscribing to the same key expression. A time series store that receives a digest from a remote replica will identify the updates it is missing. The storage then queries the missing updates from the remote storage and stores it. The structure of the digest will be different, tailored to handling time-series values.