Add support for TypedDeltaTable
pgrandjean opened this issue · 0 comments
pgrandjean commented
Frameless introduced the TypedDataset
on top of Apache Spark's Dataset API. The aim of this ticket is to create a TypedDeltaTable
on top of delta.io's DeltaTable
API. This would allow to have strongly typed merge actions, something that would look like the following:
val spark: SparkSession = ...
val updates: TypedDataset[U] = ...
val target: TypedDeltaTable[T] = TypedDataTable.unsafeForPath[T](spark, path)
val sq = target
.merge[U](updates, target('id) === updates('id))
.whenMatched(target('someSymbol) != updates('anotherSymbol))
.update(target('someSymbol) := updates('anotherSymbol))
.execute()
sq.awaitTermination()
This could be included in a new module, called frameless-delta
.