vjeantet/bitfan

[Feature request] Common unit for work with sincedb-files

Closed this issue · 4 comments

AlexAkulov commented

Hello!

You have to create your way to save the state in sincedb-file in each plugin.
Maybe its better to have common unit in logfan for saving the state that all plugins can use?

vjeantet commented

Hello, thanks for this suggestion, do you have an idea on how to model such a common unit ?

vjeantet commented

Hello @AlexAkulov I will work on this with the recent core.Storage implementation.

A Storage interface will be injected to each processor.

Processors will be able to set / get / lookup value by key.

A “sinceDB” helper could be available with this interface I’ll see.

AlexAkulov commented

Very well, thanks!

vjeantet commented

Each processor have now acces to a Storage interface (processors.IStore) which provides :

  • Get(key, namespace)
  • Set(key, namespace, value)
  • Delete(key, namespace)
  • Close()

namespace allow one processor to keep and handle data differently from one pipeline to another.
if the namespace empty, the default one will be used, and can be used share across the pipelines.

processors.SinceDB Helper introduced.

It handles all the transactions to remember files offsets over pipeline restart.
SinceDB Data is now stored in the bitfan DB (boltDB)
SinceDBPath keeps its behavior using the namespace functionality available with processors.IStore.

tips : I use https://github.com/br0xen/boltbrowser to inspect/edit the bitfan botlt BD

bitfanDB is located in .bitfan/bitfan.bold.db

see PR #58