GoogleCloudDataproc/spark-bigquery-connector

Indirect write drops policy tags

khaledh opened this issue · 2 comments

We noticed that while this issue is resolved for direct writes (#1028), it still happens when we use indirect writes; the policy tags are dropped after the indirect write. We also noticed that non-nullable columns are created as nullable when using indirect write. We're trying to work around this issue, but we're hoping it would be fixed in the connector itself.

We tried both parquet and avro intermediate formats.

(Original comment: #1028 (comment))

I updated the title/description to remove the issue about nullability. It turns out to be an issue on our side. The policy tags, however, are still not populated as expected with indirect writes.