Carve events into windows by count and timeout

[garthk]

G’day!

I trust this’ll be a simple pointer to the documentation I somehow missed rather than needing to reverse-engineer the Flow.Window protocol:

How do I carve a stream of events into windows of maximum size x but with arrival time separated by no more than y? I’d like to send many at a time, but fewer than ten if the oldest has been waiting longer than I’d like.

[josevalim]

You can do this:

      Window.global()
      |> Window.trigger_every(10)
      |> Window.trigger_periodically(5, :second)

But notice they are independent. If you reach the trigger because of a count, it won't reset the periodic timeouts. If you want more control, you can use message-based triggers with Process.send_after to control your own timer and emit_and_reduce to control the batch size.

Also note that what you want to achieve is provided out of the box by broadway: https://github.com/plataformatec/broadway