samelamin/spark-bigquery

Big query export from GCP to AWS S3 using Spark

Closed this issue · 2 comments

Hi Folks,

I would like to pull data from one of the tables in big query (GCP) to AWS S3. Both are different clusters. Is it possible to pull data from GCP to AWS, using Spark ?

Yes, this works well. I have done this. Just follow the documentation and use latest jars (com.google.cloud.bigdataoss) 2.0.0-RC2.
It would work like charm.

Yup I also have done this before, assuming spark has access via IAM roles etc then you are good to go