AbsaOSS/spark-hats

Select an element from an array of arrays

Zejnilovic opened this issue · 0 comments

Background

If you have an array of arrays select doesn't work

Feature

Be able to select a column from array of arrays

Example [Optional]

scala> res0.printSchema
root
 |-- id: long (nullable = false)
 |-- name: string (nullable = true)
 |-- items: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- itemid: string (nullable = true)
 |    |    |-- qty: integer (nullable = false)
 |    |    |-- price: double (nullable = false)
 |    |    |-- payments: array (nullable = true)
 |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |-- payid: string (nullable = true)
 |    |    |    |    |-- amount: double (nullable = false)

scala> res0.selectFromArray(col("items.payments.payid"))