/spark-template

Template project for a Spark application, used for a course on Spark

Primary LanguageScala

What is this?

This is a template project for a Spark application, created for use in a Spark course.

The Main object provides a main method which sets up your SparkContext sc (provided you set up the appName and master). After the SparkContext is created, you can interact with your cluster via this context.

It also provides a SampleTest class which gives an example of how you can create a test using a SharedSparkContext (courtesy of Holden Karau)

How to run

  • sbt assembly
  • spark-submit target/scala-2.12/spark-template-assembly-1.0.jar