itsmesrds/spark-kubernetes-demo

Spark on Kubernetes for Demo

Spark Demo on Kubernetes

###Project Summary

• brightics-on-kubernetes is a project which tries to run Brightics on Kubernetes.
  Brightics is a big data analytics platform which is based on spark-1.4.0.

• spark-demo is a demo project for a real world example. (for HDFS)
  

• spark-kubernetes-demo is a demo project for a real world example. (for S3)
  

• spark-kubernetes-master is based on
  https://github.com/kubernetes/kubernetes/tree/master/examples/spark

• spark-kubernetes-release-1.1 is based on
  https://github.com/kubernetes/kubernetes/tree/release-1.1/examples/spark

---

###Before you start, below configuration setting is needed

• If you want to run it on Local VMs via Vagrant

export KUBERNETES_PROVIDER=vagrant
export KUBERNETES_MEMORY=2048
export NUM_MINIONS=4

• If you want to run it on AWS

export KUBERNETES_PROVIDER=aws
export MASTER_SIZE=t2.small
export MINION_SIZE=t2.small
export NUM_MINIONS=4

###You can watch the demo video for Starting Spark on Kubernetes

• Demo video is based on spark-kubernetes-release-1.1 : https://asciinema.org/a/35027