/LearningSpark

Scala examples for learning to use Spark

Primary LanguageScalaMIT LicenseMIT

The LearningSpark Project

This project contains snippets of Scala code for illustrating various Apache Spark concepts. It is intended to help you get started with learning Apache Spark (as a Scala programmer) by providing a super easy on-ramp that doesn't involve Unix, cluster configuration, building from sources or installing Hadoop. Many of these activities will be necessary later in your learning experience, after you've used these examples to achieve basic familiarity.

It is intended to accompany a number of posts on the blog A River of Bytes.

Dependencies

The project was created with IntelliJ Idea 14 Community Edition, JDK 1.7, Scala 2.11.2 and Spark 1.3.0 on Windows 8.

Versions of these examples for other configurations (older versions of Scala and Spark) can be found in various branches.

Examples

The examples can be found under src/main/scala. The best way to use them is to start by reading the code and its comments. Then, since each file contains an object definition with a main method, run it and consider the output. Relevant blog posts and StackOverflow answers are listed below.

Package File What's Illustrated
Ex1_SimpleRDD How to execute your first, very simple, Spark Job. See also An easy way to start learning Spark.
Ex2_Computations How RDDs work in more complex computations. See also Spark computations.
Ex3_CombiningRDDs Operations on multiple RDDs
Ex4_MoreOperationsOnRDDs More complex operations on individual RDDs
Ex5_Partitions Explicit control of partitioning for performance and scalability.
Ex6_Accumulators How to use Spark accumulators to efficiently gather the results of distributed computations.
hiveql Using HiveQL features in a HiveContext. See the local README.md in that directory for details.
special CustomPartitioner How to take control of the partitioning of an RDD.
special HashJoin How to use the well known Hash Join algorithm to join two RDDs where one is small enough to entirely fit in the memory of each partition. See also this question on StackOverflow
special PairRDD How to operate on RDDs in which the underlying elements are pairs.
dataframe A range of DataFrame examples -- see the local README.md in that directory for details.
experiments Experimental examples that may or may not lead to anything interesting.
sql OutputJSON Write the result of a query against JSON back out as JSON text. This functionality is built into Spark 1.2.0, but the example was wrtitten to answer this question on StackOverflow in the days of Spark 1.1.0.
sql JSONSchemaInference Examples of how Spark SQL infers a schema for a file of JSON documents, including multiple cases of schema conflict.
sql UDF How to use Scala "user defined functions" (UDFs) in Spark SQL. See this question on StackOverflow. Note: Due to an apparent regression in Spark 1.3.0, the last couple of examples fail -- reported as SPARK-6054
sql CustomRelationProvider How to use the external data source provider for simple integration with an external database engine. See the blog post External Data Sources in Spark 1.2.0.
sql RelationProviderFilterPushdown More advanced integration using the external data source API, enabling filter and projection pushdown. See the blog post Filtering and Projection in Spark SQL External Data Sources.
sql ExternalNonRectangular An illustration that the Spark SQL query compiler doesn't make much use of the above pushdown possibilities in the presence of a non-rectangular Schema, like that inferred from JSON data.
streaming FileBased Streaming from a sequence of files.
streaming QueueBased Streaming from a queue.
streaming Accumulation Accumulating stream data in a single RDD.
streaming Windowing Maintaining a sliding window on the most recent stream data.

Additional Scala code is "work in progress".