datamechanics/delight

Need support for Apache Yunikorn Integration support

georgehu0815 opened this issue · 3 comments

georgehu0815 commented

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
We need add logging to Apache Yunikorn Scheduler for Spark on Kubernetes support.
This spark scheduler was deployed to Azure AKS cluster.

Describe the solution you'd like
A clear and concise description of what you want to happen.
Add logging to Apache YuniKorn job submission.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Spark history server
Additional context
Add any other context or screenshots about the feature request here.

YuniKorn Scheduler
https://yunikorn.apache.org/docs/
Deploy to k8s
https://yunikorn.apache.org/docs/developer_guide/deployment

georgehu0815 commented

.\spark-submit --master k8s://http://localhost:8001 --deploy-mode cluster --name spark-pi-delight1
--packages co.datamechanics:delight_2.12:latest-SNAPSHOT --repositories https://oss.sonatype.org/content/repositories/snapshots
--conf spark.delight.accessToken.secret=e68*****3d1af --conf spark.extraListeners=co.datamechanics.delight.DelightListener
--class org.apache.spark.examples.SparkPi --conf spark.app.name=mysparkapp-delight1
--conf spark.executor.instances=1 --conf spark.kubernetes.namespace=spark-test
--conf spark.kubernetes.executor.request.cores=1 --conf spark.kubernetes.container.image=docker.io/bochuxt/sparkdockerhisgory:3.1.1
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark file:///spark/spark-examples_2.12-3.3.1.jar

georgehu0815 commented

Looks like it do support spark submit on Apache Yunikorn Scheduler

georgehu0815 commented

.\spark-submit --class org.apache.spark.examples.SparkPi
--master 'local[2]' --name spark-pi-local
--conf spark.delight.appNameOverride=pi-localhost --conf spark.delight.url=http://127.0.0.1:9000/
--conf spark.delight.collector.url=http://127.0.0.1:9000/collector/ --packages co.datamechanics:delight_2.12:latest-SNAPSHOT
--repositories https://oss.sonatype.org/content/repositories/snapshots --conf spark.delight.accessToken.secret=e6************************3d1af
--conf spark.extraListeners=co.datamechanics.delight.DelightListener ../examples/jars/spark-examples_2.12-3.3.1.jar
10