KFP version: 1.7.0+
Kubernetes version: 1.17+
Orchestrate Spark Jobs using Kubeflow pipelines
Install kubeflow pipelines standalone or full kubeflow
for standalone kubeflow pipelines installation
https://www.kubeflow.org/docs/components/pipelines/installation/
for full kubeflow installation
https://www.kubeflow.org/docs/started/installing-kubeflow/
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator#installation
Create Spark Service Account and add permissions
kubectl apply -f ./scripts/spark-rbac.yaml
Run the notebok kubeflow-pipeline.ipynb
![image](https://github.com/images/central-ui.png)
![image](https://github.com/images/pipelines-ui.png)
Upload the spark_job_pipeline.yaml file
![image](https://github.com/images/upload-pipeline.png)
![image](https://github.com/images/create-run.png)
Start Pipeline add service account spark-sa
![image](https://github.com/images/start_run.png)
Wait till the execution is finished. check the print-message
logs to view the result
![image](https://github.com/images/final-output.png)