- example codes for spark sql data wrangling
how to hands on lab this project
- install java (jdk, not jre)
- install intellij
- install scala, sbt plugin on intellij
- create sbt project & set jdk
- set scalaVersion 2.11.* on build.sbt file (= nowaday newest scala version of spark)
- set libraryDependencies on build.sbt file (scala_sdk, spark_core, spark_sql, spark_hive)
- sbt build : intellij > View > Tool Windows > sbt > refresh
- set FileAppender on /src/main/resources/log4j.properties file to mute console log messages