spark-on-kubernetes-helm

Spark on Kubernetes infrastructure Helm charts repo

Spark on Kubernetes Platform Design concept

Contents:

Refs:

Motivation and benefits

Running Spark on Kubernetes is available since Spark 2.3.0 release on February 28, 2018. Now it is 2.4.3 and still lacks much comparing to the well known Yarn setups on Hadoop-like clusters.

Corresponding to the official docs user is able to run Spark on Kubernetes via spark-submit CLI script. And actually it is the only in-built into Apache Spark Kubernetes related capability along with some config options. Debugging proposal from Apache docs is too poor to use it easily and available only for console based tools. Schedulers integration is not available either, which makes it too tricky to setup convenient pipelines with Spark on Kubernetes out of the box. Yarn based Hadoop clusters in turn has all the UIs, Proxies, Schedulers and APIs to make your life easier.

On the other hand the usage of Kubernetes clusters in opposite to Yarn ones has definite benefits (July 2019 comparison):

Pricing. Comparing the similar cluster setups on Azure Cloud shows that AKS is about 35% cheaper than HDInsight Spark.
Scaling. Kubernetes cluster in Cloud support elastic autoscaling with many cool related features alongside, eg: Nodepools. Scaling of Hadoop clusters is far not as fast though, can be done either manually or automatically (on July 2019 was in preview).
Integrations. You can run any workloads in Kubernetes cluster wrapped into the Docker container. But do you know anyone who has ever written Yarn App in the modern world?
Support. You don't have a full control over the cluster setup provided by Cloud and usually there are no latest versions of software available for months after the release. With Kubernetes you can build image on your own.
Other Kuebernetes pros. CI/CD with Helm, Monitoring stacks ready for use in-one-button-click, huge popularity and community support, good tooling and of course HYPE.

All that makes much sense to try to improve Spark on Kubernetes usability to take the whole advantage of modern Kubernetes setups in use.

Design concept

The heart of all the problems solution is Apache Livy. Apache Livy is a service that enables easy interaction with a Spark cluster over a REST interface. It is supported by Apache Incubator community and Azure HDInsight team, which uses it as a first class citizen in their Yarn cluster setup and does many integrations with it. Watch Spark Summit 2016, Cloudera and Microsoft, Livy concepts and motivation for the details.

The cons is that Livy is written for Yarn. But Yarn is just Yet Another resource manager with containers abstraction adaptable to the Kubernetes concepts. Livy is fully open-sourced as well, its codebase is RM aware enough to make Yet Another One implementation of it's interfaces to add Kubernetes support. So why not!? Check the WIP PR with Kubernetes support proposal for Livy.

The high-level architecture of Livy on Kubernetes is the same as for Yarn.

Livy server just wraps all the logic concerning interaction with Spark cluster and provides simple REST interface.

[EXPAND] For example, to submit Spark Job to the cluster you just need to send `POST /batches` with JSON body containing Spark config options, mapped to `spark-submit` script analogous arguments.

$SPARK_HOME/bin/spark-submit \
    --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port> \
    --deploy-mode cluster \
    --name spark-pi \
    --class org.apache.spark.examples.SparkPi \
    --conf spark.executor.instances=5 \
    --conf spark.kubernetes.container.image=<spark-image> \
    local:///path/to/examples.jar
 
# is equals to
 
curl -H 'Content-Type: application/json' -X POST \
  -d '{
        "name": "SparkPi",
        "className": "org.apache.spark.examples.SparkPi",
        "numExecutors": 5,
        "conf": {
          "spark.kubernetes.container.image": "<spark-image>"
        },
        "file": "local:///path/to/examples.jar"
      }' "http://livy.endpoint.com/batches"

Under the hood Livy parses POSTed configs and does spark-submit for you, bypassing other defaults configured for the Livy server, eg.: --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port> value.

After the job submission Livy discovers Spark Driver Pod scheduled to the Kubernetes cluster with Kubernetes API and starts to track its state, cache Spark Pods logs and details descriptions making that information available through Livy REST API, builds routes to Spark UI, Spark History Server, Monitoring systems with Kubernetes Ingress resources, Nginx Ingress Controller in particular and displays the links on Livy Web UI.

Providing REST interface for Spark Jobs orchestration Livy allows any number of integrations with Web/Mobile apps and services, easy way of setting up flows via jobs scheduling frameworks.

Livy has in-built lightweight Web UI, which makes it really competitive to Yarn in terms of navigation, debugging and cluster discovery.

Livy supports interactive sessions with Spark clusters allowing to communicate between Spark and application servers, thus enabling the use of Spark for interactive web/mobile applications. Using that feature Livy integrates with Jupyter Notebook through Sparkmagic kernel out of box giving user elastic Spark exploratory environment in Scala and Python. Just deploy it to Kubernetes and use!

On top of Jupyter it is possible to set up JupyterHub, which is a multi-user Hub that spawns, manages, and proxies multiple instances of the single-user Jupyter notebook servers. Follow the video PyData 2018, London, JupyterHub from the Ground Up with Kubernetes - Camilla Montonen to learn the details of the implementation. JupyterHub provides a way to setup auth through Azure AD with AzureAdOauthenticator plugin as well as many other Oauthenticator plugins.

Monitoring setup of Kubernetes cluster itself can be done with Prometheus Operator stack with Prometheus Pushgateway and Grafana Loki using a combined Helm chart, which allows to do the work in one-button-click. Learn more about the stack from videos:

The overall monitoring architecture solves pull and push model of metrics collection from the Kubernetes cluster and the services deployed to it. Prometheus Alertmanager gives an interface to setup alerting system.

With the help of JMX Exporter or Pushgateway Sink we can get Spark metrics inside the monitoring system. Grafana Loki provides out-of-box logs aggregation for all Pods in the cluster and natively integrates with Grafana. Using Grafana Azure Monitor datasource and Prometheus Federation feature you can setup complex global monitoring architecture for your infrastructure.

Installation guide

TODO

Known issues and tips