Exception when writing an empty dataframe to cdm using Spark CDM connector
Closed this issue · 1 comments
absognety commented
My code for writing to cdm (connector version 0.18.2)
(df.write.format("com.microsoft.cdm")
.option("storage", storage_account_name)
.option("manifestPath", container_name + "/" + cdm_folder_name + "/default.manifest.cdm.json")
.option("entity", entity_name)
.option("appId", app_id)
.option("appKey", app_key)
.option("tenantId", tenant_id)
.mode('overwrite')
.option("columnHeaders", False)
.save())
This is the exception:
An error occurred while calling o1896.save. : java.util.concurrent.CompletionException: java.lang.Exception: PersistenceLayer | Could not convert 'default.manifest.cdm.json'. Reason 'null'. | loadDocumentFromPathAsync at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606) at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1596) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: java.lang.Exception: PersistenceLayer | Could not convert 'default.manifest.cdm.json'. Reason 'null'. | loadDocumentFromPathAsync at com.microsoft.cdm.utils.CDMCallback$.apply(CDMUtils.scala:74) at com.microsoft.commondatamodel.objectmodel.utilities.logger.Logger.log(Logger.java:136) at com.microsoft.commondatamodel.objectmodel.utilities.logger.Logger.error(Logger.java:109) at com.microsoft.commondatamodel.objectmodel.persistence.PersistenceLayer.lambda$loadDocumentFromPathAsync$0(PersistenceLayer.java:249) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) ... 5 more
Let me know if you need anything else
absognety commented
We fixed this ourselves, default.manifest.cdm.json
got corrupted somehow so, we deleted the data and re-run our pipeline which generated manifest file as expected, closing this.