/SparkStudy

example codes for spark sql data wrangling

Primary LanguageScala

SparkStudy

  • example codes for spark sql data wrangling

how to hands on lab this project

  1. install java (jdk, not jre)
  2. install intellij
  3. install scala, sbt plugin on intellij
  4. create sbt project & set jdk
  5. set scalaVersion 2.11.* on build.sbt file (= nowaday newest scala version of spark)
  6. set libraryDependencies on build.sbt file (scala_sdk, spark_core, spark_sql, spark_hive)
  7. sbt build : intellij > View > Tool Windows > sbt > refresh
  8. set FileAppender on /src/main/resources/log4j.properties file to mute console log messages