Big query export from GCP to AWS S3 using Spark
Closed this issue · 2 comments
ChitturiPadma commented
Hi Folks,
I would like to pull data from one of the tables in big query (GCP) to AWS S3. Both are different clusters. Is it possible to pull data from GCP to AWS, using Spark ?
ameyamahajan commented
Yes, this works well. I have done this. Just follow the documentation and use latest jars (com.google.cloud.bigdataoss) 2.0.0-RC2.
It would work like charm.
samelamin commented
Yup I also have done this before, assuming spark has access via IAM roles etc then you are good to go