AbsaOSS/spark-hofs

dealing with structs within transform

Closed this issue · 2 comments

Thank you for adding this library, helpful to go via api route.

I am struggling to use transform for array<struct<cat:Long,ctg:string>>

transform(
                    col(qualifiedFieldName),
                    x=>{
                      x.cat
                    },
                    "el"
                  )

this throws compile time exception

transform(
                    col(qualifiedFieldName),
                    x=>{
                      col("el.cat")
                    },
                    "el"
                  )

throws org.apache.spark.sql.AnalysisException: cannot resolve '`el.aid`' given input columns

Accessing elements of a struct inside an array is a common use case. Here is an example:

      transform(
        col(qualifiedFieldName),
        x => {
          x.getField("cat")
        },
        "el"
      )