[Bug] Spark 3.3: Reading a table gives: java.util.NoSuchElementException: None.get

[kecheung]

Did you read the pinned issues and search the error message?

Yes, but I didn't find the answer.

Summary of issue

I am reading a table and get the following exception.

Error stack trace

None.get
java.util.NoSuchElementException: None.get
    at scala.None$.get(Option.scala:529)
    at scala.None$.get(Option.scala:527)
    at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Utils$.loadV2Source(DataSourceV2Utils.scala:129)
    at org.apache.spark.sql.DataFrameReader.$anonfun$load$1(DataFrameReader.scala:209)
    at scala.Option.flatMap(Option.scala:271)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:207)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:171)

Platform name

Databricks

Spark version

Spark 3.3

CDM jar version

spark3.3-1.19.5

What is the format of the data you are trying to read/write?

.csv

[kecheung]

It's a Spark bug, and we already fixed it in Synapse. There is nothing wrong with the connector. As mentioned in the repo, the connector is designed for Synapse.

For other Spark providers, ask them to backport SPARK-41344 to Spark 3.3. The open source code was fixed for Spark 3.4+.