TU-Berlin-DIMA/scotty-window-processor

[Bug] Incorrect Slices of Sliding Window depending on Slide Value

Closed this issue · 1 comments

Scotty's StreamSlicer starts a new slice based on the window edge that is returned by the function assignNextWindowStart.

public long assignNextWindowStart(long recordStamp) {
return recordStamp + getSlide() - (recordStamp) % getSlide();
}

This does not work for sliding windows whose slide value is not a multiple of the window size, for instance, a sliding window with size = 10 and slide = 3.
In these cases, the values of the window starts and the window ends are not identical.
The function only returns the next window start which is based on the slide value, but does not return the next window end.
Thus, the slices do not end when a window ends, leading to incorrect window results.

This issue occurred during the migration to the open-source project. It is not a limitation of Scotty.
The concept of modeling a sliding window by two tumbling windows is correctly described in the paper Scotty: General and Efficient Open-source Window Aggregation for Stream Processing Systems (TODS 2021).