ddm-spark

Spark example and homework code for the "Distributed Data Management" lecture.

Setup instructions

This is a basic scala project that includes spark dependencies and necessary build-configuration to build a jar that can be submitted to a spark installation via spark-submit.

Note that the project is only compatible with Java <= 11. We recommend that you use Java 11 to build and run this project.

Checkout the project

git clone git@github.com:UMR-Big-Data-Analytics/ddm-spark.git

Install IntelliJ

Select the Scala and sbt plugins to be installed during the installation; otherwise you need to install them later

Import the project into IntelliJ

Start Intellij
Select Open or Import
Select the ddm-spark/build.sbt file and click OK
Click Open as Project (IntelliJ should automatically recognize the sbt project nature; note that the initialization might take a while for missing downloads and indexing; if IntelliJ does not automatically download the depencencies, you can open the sbt tab on the right edge of the screen and click Reload all sbt projects)

Load the homework data

Download the tpch.zip from https://hpi.de/fileadmin/user_upload/fachgebiete/naumann/lehre/WS2017/DDA/TPCH.zip
Extract the zipped content into ddm-spark/data, which should create the folder ddm-spark/data/TPCH with 7 csv files

Run the tutorial

Switch to your IntelliJ window and open ddm-spark/src/main/scala/de/ddm in the project tree view
Right-click DDMSpark.scala and click Run DDMSpark

Solve the DDM homework

Remove the Tutorial and LongestCommonSubstring calls
Implement the Sindy algorithm

Build your final jar file

Click Run->Edit Configurations->+->SBT Task
Enter a name, such as DDMSpark assembly
Enter task clean assembly
Click OK
You can now switch to and run the DDMSpark assembly target in the top right Intellij bar
Find your fat-jar in ddm-spark/target/scala-2.12/DDMSpark-assembly-0.1.jar

Some further notes for cluster submits

Modify the settings in the build.sbt to match the spark installation (Relevant settings are both the scalaVersion parameter and the specific versions of all the spark packages (for example in libraryDependencies += org.apache.spark %% spark-core % x replace x with the version of the spark installation, where you want to execute the code). If the scala version and the spark version do not exactly match those of the spark installation, you will encounter errors while executing the jar that are not really helpful in determining the cause.