/example-spark

Example Spark, Spark Streaming and Spark SQL

Primary LanguageScala

Spark, Spark Streaming and Spark SQL unit testing strategies

Build Status Coverage Status

Features

  • ClockWrapper for efficient clock management in Spark Streaming jobs.
  • Base traits for testing Spark, Spark Streaming and Spark SQL to eliminate boilerplate code.
  • Sample applications to show how to make your code testable.
  • All tests can be run or debugged directly from IDE, or using SBT.
  • All test fixtures are prepared as in-memory data structures.
  • SBT is configured to avoid problems with multiple Spark contexts in the same JVM SPARK-2243.
  • Logger is configured, uncomment specific loggers in logback-test.xml if needed.

References