3.5 with delta - NakedLocalFileSystem$NakedLocalFileStatus cannot be cast to FileStatusWithMetadata

[chris-twiner]

running the ExtensionsDeltaTest with 3.5 snap causes this to be thrown from many of the tests that do not use onlyWithExistingSession / wrapWithExistingSession (i.e. those to run on Databricks / notebooks).

java.lang.ClassCastException: com.globalmentor.apache.hadoop.fs.NakedLocalFileSystem$NakedLocalFileStatus cannot be cast to org.apache.spark.sql.execution.datasources.FileStatusWithMetadata

same delta version on 3.4 per latter #25 builds have no issue. (some of them also failed with the npe issue 629)

[chris-twiner]

as that's new for 3.5 an expected new delta would be needed https://issues.apache.org/jira/browse/SPARK-43039

[chris-twiner]

also

java.lang.NoSuchMethodError: org.apache.spark.sql.types.StructType.toAttributes()Lscala/collection/Seq;

	at org.apache.spark.sql.delta.files.TransactionalWrite.getStatsSchema(TransactionalWrite.scala:262)
	at org.apache.spark.sql.delta.files.TransactionalWrite.getStatsSchema$(TransactionalWrite.scala:252)
	at org.apache.spark.sql.delta.OptimisticTransaction.getStatsSchema(OptimisticTransaction.scala:137)
	at org.apache.spark.sql.delta.files.TransactionalWrite.getOptionalStatsTrackerAndStatsCollection(TransactionalWrite.scala:294)
	at org.apache.spark.sql.delta.files.TransactionalWrite.getOptionalStatsTrackerAndStatsCollection$(TransactionalWrite.scala:286)
	at org.apache.spark.sql.delta.OptimisticTransaction.getOptionalStatsTrackerAndStatsCollection(OptimisticTransaction.scala:137)
	at org.apache.spark.sql.delta.files.TransactionalWrite.writeFiles(TransactionalWrite.scala:352)