[BUG] Service Account not working different namespace

[devscheffer]

Description

I use the helmchart of spark operator, it is deployed at the namespace spark-operator I configure on the helmrelease sparkJobNamespaces: spark-jobs that is the namespace where I want to run the jobs.
However, I'm getting this error

Name: "pyspark-pi", Namespace: "spark-jobs"
from server for: "STDIN": sparkapplications.sparkoperator.k8s.io "pyspark-pi" is forbidden: User "system:serviceaccount:spark-jobs:spark-sa" cannot get resource "sparkapplications" in API group "sparkoperator.k8s.io" in the namespace "spark-jobs"

[ChenYi015]

@devscheffer Could you provide detailed information about how you install the helm chart? Is this service account spark-sa created by helm or by yourself?

[devscheffer]

it is created by the helm.

---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  labels:
    app: spark-operator
  name: spark-operator
  namespace: spark-operator
spec:
  chart:
    spec:
      chart: spark-operator
      reconcileStrategy: ChartVersion
      sourceRef:
        kind: HelmRepository
        name: spark-operator
      version: 1.4.0
  interval: 5m0s
  releaseName: spark-operator
  values:
    image:
      repository: docker.io/kubeflow/spark-operator
      pullPolicy: IfNotPresent
      tag: ""
    rbac:
      create: false
      createRole: true
      createClusterRole: true
      annotations: {}
    serviceAccounts:
      spark:
        create: true
        name: "spark-sa"
        annotations: {}
      sparkoperator:
        create: true
        name: "spark-operator-sa"
        annotations: {}
    sparkJobNamespaces:
      - spark-operator
      - team-1
    webhook:
      enable: true
      port: 443
      portName: webhook
      namespaceSelector: ""
      timeout: 30
    metrics:
      enable: true
      port: 10254
      portName: metrics
      endpoint: /metrics
      prefix: ""  
    tolerations:
      - key: "CriticalAddonsOnly"
        operator: "Exists"
        effect: "NoSchedule"

It works when I do manually through the terminal however when I execute from airflow I get this error
from server for: "STDIN": sparkapplications.sparkoperator.k8s.io "pyspark-pi2" is forbidden: User "system:serviceaccount:team-1:spark-sa" cannot get resource "sparkapplications" in API group "sparkoperator.k8s.io" in the namespace "team-1"

here is the task in airflow

spark_kpo = KubernetesPodOperator(
        task_id="kpo",
        name="spark-app-submission",
        namespace=namespace,
        image="bitnami/kubectl:1.28.11",
        cmds=["/bin/bash", "-c"],
        arguments=[f"echo '{spark_app_manifest_content}' | kubectl apply -f -"],
        in_cluster=True,
        get_logs=True,
        service_account_name=service_account_name,
        on_finish_action="keep_pod",
    )
    ```

[ChenYi015]

@devscheffer The service account spark-sa actually does not have any permissions for SparkApplication, and it is used by spark driver pods. If you want to submit SparkApplication in airflow, you can configure the service account name to spark-operator-sa in KubernetesPodOperator instead. Or you can create a ServiceAccount manually and grant it with all permissions to SparkApplication.