typelevel/frameless

Add support for TypedDeltaTable

pgrandjean opened this issue · 0 comments

Frameless introduced the TypedDataset on top of Apache Spark's Dataset API. The aim of this ticket is to create a TypedDeltaTable on top of delta.io's DeltaTable API. This would allow to have strongly typed merge actions, something that would look like the following:

val spark: SparkSession = ...
val updates: TypedDataset[U] = ...
val target: TypedDeltaTable[T] = TypedDataTable.unsafeForPath[T](spark, path)

val sq = target
  .merge[U](updates, target('id) === updates('id))
  .whenMatched(target('someSymbol) != updates('anotherSymbol))
  .update(target('someSymbol) := updates('anotherSymbol))
  .execute()

sq.awaitTermination()

This could be included in a new module, called frameless-delta.