Impersonate Service Account

[lcaggio]

I am writing a pyspark script on GCP Dataproc to:

• impersonate a service account
• read data from GCS (using the credential of the impersonated SA)
• write data to BigQuery (using the credential of the impersonated SA)

The Dataproc service account has AccessTokenCreator role on the service account to be impersonated (delegated_sa), the delegated_sa has access to GCS and BQ.

Script

...
spark = SparkSession.builder
.appName("Read CSV from GCS and Write to BigQuery")
.config('spark.hadoop.fs.gs.auth.impersonation.service.account', delegated_sa)
.config('gcpImpersonationServiceAccount', delegated_sa)
.getOrCreate()
...
data = spark.read.format("csv")
.schema(schema)
.load(csv)
...
data.write.format('bigquery')
.option('table', dataset_table)
.mode('append')
.save()
...

Error

        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.GatewayConnection.run(GatewayConnection.java:238)
        at java.lang.Thread.run(Thread.java:750)
Caused by: com.google.cloud.spark.bigquery.repackaged.com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
GET https://www.googleapis.com/bigquery/v2/projects/dataproc/datasets/tables/customers?prettyPrint=false
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Access Denied: Tabledataproc:customers: Permission bigquery.tables.get denied on tabledataproc:customers (or it may not exist).",
    "reason" : "accessDenied"
  } ],
  "message" : "Access Denied: Tabledataproc:dataproc_out.customers: Permission bigquery.tables.get denied on tabledataproc:customers (or it may not exist).",
  "status" : "PERMISSION_DENIED"
}
        at com.google.cloud.spark.bigquery.repackaged.com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
        at com.google.cloud.spark.bigquery.repackaged.com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
        at com.google.cloud.spark.bigquery.repackaged.com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
        at com.google.cloud.spark.bigquery.repackaged.com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:428)
        at com.google.cloud.spark.bigquery.repackaged.com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
        at com.google.cloud.spark.bigquery.repackaged.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
        at com.google.cloud.spark.bigquery.repackaged.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
        at com.google.cloud.spark.bigquery.repackaged.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
        at com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.getTable(HttpBigQueryRpc.java:284)
        ... 45 more

Note

• The service account impersonification for GCS works properly
• If I do service account credentials over json key (.config("credentials", BASE64)) instead of impersonification, it works properly

[lcaggio]

Closing, I was using an old version of the bigquery connector ... Using the spark-bigquery-with-dependencies_2.12-0.36.1.jar it works with no issues.