[Bug] Spark 3.3: Reading a table gives: java.util.NoSuchElementException: None.get
Closed this issue · 1 comments
kecheung commented
Did you read the pinned issues and search the error message?
Yes, but I didn't find the answer.
Summary of issue
I am reading a table and get the following exception.
Error stack trace
None.get
java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:529)
at scala.None$.get(Option.scala:527)
at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Utils$.loadV2Source(DataSourceV2Utils.scala:129)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$1(DataFrameReader.scala:209)
at scala.Option.flatMap(Option.scala:271)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:207)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:171)
Platform name
Databricks
Spark version
Spark 3.3
CDM jar version
spark3.3-1.19.5
What is the format of the data you are trying to read/write?
.csv
kecheung commented
It's a Spark bug, and we already fixed it in Synapse. There is nothing wrong with the connector. As mentioned in the repo, the connector is designed for Synapse.
For other Spark providers, ask them to backport SPARK-41344 to Spark 3.3. The open source code was fixed for Spark 3.4+.