Azure/spark-cdm-connector

[Issue] Problem datetime format read in spark connector

Opened this issue · 0 comments

Did you read the pinned issues and search the error message?

No, but I will read and search it now before creating an issue.

Summary of issue

I have received data BC365 F&O a data source type Common Data Model stored in Azure Data Storage Gen2 and
using pyspark read data format com.microsoft.cdm, It can't pares datetime datatype see below.

Please help me find a solution to this problem.

csv format
image

pyspark read with connector
image

Error stack trace

No response

Platform name

Azure Synapse

Spark version

3.3

CDM jar version

com.microsoft.cdm.BuildInfo.version

What is the format of the data you are trying to read/write?

.csv