SPARK

Spark Documentation

Spark Source Code Installation

Download the spark source code from here
Unzip the downloaded tar: tar -xvzf spark-2.0.1.tgz
Enter the spark directory: cd spark-2.0.1
Build the source code: ./build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0 -Phive -Phive-thriftserver -DskipTests package

Spark Shell

cd spark-2.0.1
./bin/spark-shell

This will open the spark shell where you can code in scala.

You might see the following warning when the spark shell is starting up: WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

This warning won't cause any issues unless your code depends on a binary like libsnappy.so which is present under native-hadoop library location.
To resolve this warning, include the native-hadoop library location in the cluster's LD_LIBRARY_PATH environment variable:

export LD_LIBRARY_PATH=/opt/cloudera/parcels/CDH/lib/hadoop/lib/native:$LD_LIBRARY_PATH

SBT (Scala Build Tool)

SBT is a tool to build your spark application (coded in scala) into a jar. The built jar can then be run using spark-submit.

Download SBT from here
Unzip the downloaded tar: tar -xvzf <file.tgz>
Add sbt executable location to PATH variable: export PATH=/data/home/devbld/sbt-launcher-packaging-0.13.13/bin:$PATH

NOTE - Don't use the SBT that comes with spark installation. There are certain bugs in there.

Self-Contained Spark Application

Let's suppose that we have a created a spark application in scala by the name of SimpleApp.scala. To build it, we will be using SBT.
SBT requires a configuration file. Let's call it simple.sbt with the following content:

name := "Simple Project"

version := "1.0"

scalaVersion := "2.11.7"

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.1"

For SBT to build our spark code correctly and produce a jar, we need to put SimpleApp.scala and simple.sbt in a typical directory structure:

$ find .
.
./simple.sbt
./src
./src/main
./src/main/scala
./src/main/scala/SimpleApp.scala
./project
./target

We will use sbt to package our spark application

$ sbt package

This will generate a jar under ./target/scala-2.11

We will use spark-submit to run the application