GoogleCloudDataproc/spark-bigquery-connector

Saving data using Spark BigQuery connector with ranged index

xtrmstep opened this issue · 5 comments

xtrmstep commented

Hi,
Could you, please, help with the described problem related to the connector? If this is possible at all.
The problem described in the question on stackoverflow: https://stackoverflow.com/questions/75142769/saving-data-using-spark-bigquery-connector-with-ranged-index

vishalkarve15 commented

Added support for this.

xtrmstep commented

@vishalkarve15 could you, please, advise about the documentation link to understand how exactly is possible to use it?

vishalkarve15 commented

It will be present in the next release. For now, you can look at the partitionField option in https://github.com/GoogleCloudDataproc/spark-bigquery-connector/blob/master/README-template.md

vishalkarve15 commented

If you want to use it right away, you can build the latest commit.

  • 👍1
davidrabinowitz commented

You can also try one of the nightly builds (based on your spark version):

  • gs://spark-lib-nightly-snapshots/spark-2.4-bigquery-0.0.20230808.jar
  • gs://spark-lib-nightly-snapshots/spark-3.1-bigquery-0.0.20230808.jar
  • gs://spark-lib-nightly-snapshots/spark-3.2-bigquery-0.0.20230808.jar
  • gs://spark-lib-nightly-snapshots/spark-3.3-bigquery-0.0.20230808.jar
  • gs://spark-lib-nightly-snapshots/spark-3.4-bigquery-0.0.20230808.jar