NVIDIA/spark-rapids

[FEA] [AUDIT] Support reading Avro with recursive schema reference

Opened this issue · 0 comments

SPARK-48939 allows Spark 4.0 to now read Avro input where the schema is self-referential.

For instance:

{
  "type": "record",
  "name": "LongList",
  "fields" : [
    {"name": "value", "type": "long"},
    {"name": "next", "type": ["null", "LongList"]}
  ]
}

This isn't yet supported in the spark-rapids plugin's Avro reader. This might be required at some point, for parity.