saveAsBigQueryTable exception with StructType column
Opened this issue · 1 comments
andreclaudino commented
I am trying to save data using saveAsBigQueryTable
, but this is resulting in an error. I simplified the dataframe for a single column, an StructType one, this is the schema:
root
|-- contexto: struct (nullable = true)
| |-- b2wChannel: string (nullable = true)
| |-- b2wDeviceType: string (nullable = true)
| |-- b2wEPar: string (nullable = true)
| |-- b2wOpn: string (nullable = true)
| |-- b2wPid: string (nullable = true)
| |-- b2wSid: string (nullable = true)
| |-- b2wUid: string (nullable = true)
| |-- customerId: string (nullable = true)
| |-- salesSolution: string (nullable = true
When using other fields (of other types) the dataset is saved as expected, so, the error is in StructType columns.
This is the stack
[error] (run-main-0) java.lang.NoSuchMethodError: org.json4s.JsonDSL$.seq2jvalue(Lscala/collection/Traversable;Lscala/Function1;)Lorg/json4s/JsonAST$JArray;
[error] java.lang.NoSuchMethodError: org.json4s.JsonDSL$.seq2jvalue(Lscala/collection/Traversable;Lscala/Function1;)Lorg/json4s/JsonAST$JArray;
[error] at com.samelamin.spark.bigquery.converters.SchemaConverters$$anonfun$typeToJson$3.apply(SchemaConverters.scala:66)
[error] at com.samelamin.spark.bigquery.converters.SchemaConverters$$anonfun$typeToJson$3.apply(SchemaConverters.scala:66)
[error] at org.json4s.JsonDSL$JsonAssoc.$tilde(JsonDSL.scala:90)
[error] at com.samelamin.spark.bigquery.converters.SchemaConverters$.typeToJson(SchemaConverters.scala:66)
[error] at com.samelamin.spark.bigquery.converters.SchemaConverters$.com$samelamin$spark$bigquery$converters$SchemaConverters$$fieldToJson(SchemaConverters.scala:84)
[error] at com.samelamin.spark.bigquery.converters.SchemaConverters$$anonfun$1.apply(SchemaConverters.scala:87)
[error] at com.samelamin.spark.bigquery.converters.SchemaConverters$$anonfun$1.apply(SchemaConverters.scala:87)
[error] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
[error] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
[error] at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
[error] at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
[error] at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
[error] at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
[error] at com.samelamin.spark.bigquery.converters.SchemaConverters$.SqlToBQSchema(SchemaConverters.scala:87)
[error] at com.samelamin.spark.bigquery.BigQueryDataFrame.saveAsBigQueryTable(BigQueryDataFrame.scala:41)
[error] at ml.freight.prepare.CarregaPedidos$.delayedEndpoint$ml$freight$prepare$CarregaPedidos$1(CarregaPedidos.scala:24)
[error] at ml.freight.prepare.CarregaPedidos$delayedInit$body.apply(CarregaPedidos.scala:6)
[error] at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
[error] at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
[error] at scala.App$$anonfun$main$1.apply(App.scala:76)
[error] at scala.App$$anonfun$main$1.apply(App.scala:76)
[error] at scala.collection.immutable.List.foreach(List.scala:392)
[error] at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
[error] at scala.App$class.main(App.scala:76)
[error] at ml.freight.prepare.CarregaPedidos$.main(CarregaPedidos.scala:6)
[error] at ml.freight.prepare.CarregaPedidos.main(CarregaPedidos.scala)
[error] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error] at java.lang.reflect.Method.invoke(Method.java:498)
samelamin commented
The problem is most probably in transforming the struct type to the bq schema
Can you provide a sample json record of the broken file so it can help the investigation