GoogleCloudDataproc/spark-bigquery-connector

Schema mismatch error needs to be more verbose

Closed this issue · 2 comments

vishalkarve15 commented

The current error message when writing to a BigQuery table is very vague: "Destination table's schema is not compatible with dataframe's schema"

The extent of the schema mismatch is unclear and there is no indication if only the mode check (required vs nullable) needs fixing.

spark-bigquery-connector/spark-bigquery-connector-common/src/main/java/com/google/cloud/spark/bigquery/write/context/BigQueryDirectDataSourceWriterContext.java

Line 151 in 76dc211

"Destination table's schema is not compatible with dataframe's schema"));

jster1357 commented

+1 for this. Being able to see what the offending columns are would reduce overall developer friction when building pipelines.

vishalkarve15 commented

Fixed in version 0.41.0