/specifying-schema

Show different ways to specify schema with Spark + Scala, and potential issues that can occur.

Primary LanguageScala

Specifying schema examples

The purpose of this repo is to make sure that you understand how to pass in schema for different datasets.

Pre-requisites

Please make sure you have the following installed

  • Java 8
  • Scala 2.11
  • Sbt 1.1.x
  • Apache Spark 2.4 with ability to run spark-submit locally

Setup for local building and testing

  • Clone this repo
  • Build: sbt package
  • Test: sbt test

Goal

  • Fix the code such that the tests pass.