Arrow Streaming Analytics

[Max-Meldrum]

We are currently using Arrow mainly for converting Protobuf state into queryable Arrow data. I think it would be interesting to see how we could use Arrow for streaming operations and how this can be exposed in the API (e..g, Dataframes?).

As of now, the only thing we have is ArrowWindow that exposes (Arc<Schema>, Vec<RecordBatch>) per Window trigger.

Relevant references:

https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html
https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/trill-vldb2015.pdf
https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/table/concepts/dynamic_tables/

[Max-Meldrum]

Maybe we could define a ArrowStream type which would support a number of transformations.