LaggAt/DimensionTabler

update older dimension table rows

Closed this issue · 0 comments

For now we just create dimension tables once, and keep pulling new data in.
In next relase we will get rid of old dimension table entries. Before we do that, we would need to truncate the whole table to match statistics as in the proposal. So this is a core feature.

In the example on README.md the second granularity is 15', the first was 1'. We can not just boot 14 rows, that will possibly break some aggregations, and we do not want to rely on the fact, that the next granularity is a multiply of the current.

I've two solutions in mind, still need to decide which route to take from here.

  1. recalculate (stepwise) from the beginning from time to time: the easiest, as we do not need to save variable states or anything else, we just do the calculations again and boot any non-matching dimension table entry.

  2. the smart way: persist variables and just redo all related current blocks + all following. This shouldn't break any variable usage, as prior data isn't touched at all and all future data is regenerated.