Examples for Learning Spark
Examples for the Learning Spark book.
Requirements
- JDK 1.6 or higher
- Scala 2.10.3
- scala-lang.org
- Spark 1.0 sanp shot
- You can checkout spark from https://github.com/apache/spark and then run "sbt/sbt publish-local"
- Protobuf compiler
- On debian you can install with sudo apt-get install protobuf-compiler
Scala examples
You can build and run the Scala examples with sbt, just run sbt/sbt compile package run
Java examples
You can build and run the Java examples wih maven, just run mvn package mvn exec:java -Dexec.mainClass="com.oreilly.learningsparkexamples.java.[EXAMPLE]"
Python examples
From spark just run ./bin/pyspark ./src/python/[example]
Spark Submit
You can also create an assembly jar with all of the dependcies for running either the java or scala versions of the code and run the job with the spark-submit script
./sbt/sbt assembly cd $SPARK_HOME; ./bin/spark-submit --class com.oreilly.learningsparkexamples.[lang].[example] ../learning-spark-examples/target/scala-2.10/learning-spark-examples-assembly-0.0.1.jar