Arrow Streaming Analytics
Max-Meldrum opened this issue · 1 comments
We are currently using Arrow mainly for converting Protobuf state into queryable Arrow data. I think it would be interesting to see how we could use Arrow for streaming operations and how this can be exposed in the API (e..g, Dataframes?).
As of now, the only thing we have is ArrowWindow
that exposes (Arc<Schema>, Vec<RecordBatch>)
per Window trigger.
Relevant references:
https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html
https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/trill-vldb2015.pdf
https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/table/concepts/dynamic_tables/
Maybe we could define a ArrowStream
type which would support a number of transformations.