NoSuchTableException: Manifest doesn't exist. root.manifest.cdm.json #Spark3.2.1 jar #Databricks

[RushikeshGuajr]

Hi @sricheta92 @kecheung ,

I'm facing an issue while running the below mention code to create file on adls using Spark Connector 3.2.1.
Error message : NoSuchTableException: Manifest doesn't exist. root.manifest.cdm.json

We are trying to run sample code shared by you. We are using below mention config for Databricks.
Runtime version : 10.4 LTS
Library : spark-cdm-connector-assembly-synapse-spark3.2-1.19.4.jar

Could you please help us ?
Please let me know if more information is required.

Thanks in advance.

[dlpkmr98]

Hi @sricheta92 @kecheung ,

i am also facing same issue when using following sample code with spark-cdm-connector-assembly-synapse-spark3.2-1.19.4.jar

// COMMAND ----------

// Explicit write, creating an entity in a CDM folder based on a pre-defined model

// Case 2: Using an entity definition defined in a CDM model stored in ADLS

// UPLOAD CDM FILES FIRST
// To run this example, first create a /Models/Contacts folder to your demo container in ADLS gen2,
// then upload the provided Contacts.manifest.cdm.json, Person.cdm.json, Entity.cdm.json files

val birthdate= java.sql.Date.valueOf("1991-03-31");
val now = new java.sql.Timestamp(System.currentTimeMillis());
val data2 = Seq(
  Row(1,now,"Donna","Carreras",birthdate),
  Row(2,now,"Keith","Harris",birthdate),
  Row(2,now,"Carla","McGee",birthdate)
)

val schema2 = new StructType()
  .add(StructField("identifier", IntegerType))
  .add(StructField("createdTime", TimestampType))
  .add(StructField("firstName", StringType))
  .add(StructField("lastName", StringType))
  .add(StructField("birthDate", DateType))

// Create the dataframe that matches the CDM definition of the entity, Person
val df2 = spark.createDataFrame(spark.sparkContext.parallelize(data2, 1), schema2)
df2.write.format("com.microsoft.cdm")
  .option("storage", storageAccountName)
  .option("manifestPath", container + "/Data/Contacts/root.manifest.cdm.json")
  .option("entity", "Person")
  .option("entityDefinitionModelRoot", container + "/Models")
  .option("entityDefinitionPath", "/Contacts/Person.cdm.json/Person")
  .mode(SaveMode.Overwrite)
  .save()

• do we have to add something for spark 3 cdm connector lib.?. as same code is working fine with spark 2 cdm connector.
• do we have to create root.manifest.cdm.json manually ?

please suggest some workaround as i have to change the spark version from 2 to 3.
Thanks in advance!

[kecheung]

@RushikeshGuajr @dlpkmr98 The sample might be out of date. There is a behavior change in the save behavior if you are moving from Spark 2 to Spark 3. Please remove the .mode(SaveMode.Overwrite) option . You are writing the entity for the first time and because it never existed, you will get the error. I have tried the sample and it works for me with that change.

Added save mode changes to the common issues #118

It's mentioned briefly here and the default behavior is SaveMode.ErrorIfExists. https://github.com/Azure/spark-cdm-connector/blob/spark3.2/documentation/overview.md#save-mode