/net.jgp.books.spark.ch17

Spark in Action, 2nd edition - chapter 16 - exporting data, using delta lake

Primary LanguageJavaApache License 2.0Apache-2.0

This repository contains the Java labs as well as their Scala and Python ports of the code used in Manning Publication’s Spark in Action, 2nd edition, by Jean-Georges Perrin.

Spark in Action, 2nd edition – Java, Python, and Scala code for chapter 17

Chapter 17 is about exporting data.

This code is designed to work with:

  • Apache Spark v3.0.0.
  • Delta Lake v0.7.0.

Labs

Lab #100

Adapted from one of IBM's Code for Call starter kit, but leveraging the wildfires category.

Lab #200

Feeds data into Delta Lake.

Lab #210, #220

Run analytics on data stored in Delta Lake.

Lab #250

Feeds data into Delta Lake.

Lab #900

Append using primary key.

Lab #910

Export widlfire data to Elasticsearch.

Datasets

Dataset(s) used in this chapter:

Lab #100

The ExportWildfiresApp application does the following:

  1. It acquires a session (a SparkSession).
  2. It asks Spark to load (ingest) a dataset in CSV format.
  3. Spark stores the contents in a dataframe, do some transformations and export final reports as CSV data.

Running the lab in Java

For information on running the Java lab, see chapter 1 in Spark in Action, 2nd edition.

Running the lab using PySpark

Prerequisites:

You will need:

  • git.
  • Apache Spark (please refer Appendix P - 'Spark in production: installation and a few tips').
  1. Clone this project
git clone https://github.com/jgperrin/net.jgp.books.spark.ch17
  1. Go to the lab in the Python directory
cd net.jgp.books.spark.ch17/src/main/python/lab100_orders/
  1. Execute the following spark-submit command to create a jar file to our this application
spark-submit orderStatisticsApp.py

NOTE:- If you want to run delta lake examples, please use the following spark-submit command:

spark-submit --driver-class-path /tmp/jars/io.delta_delta-core_2.12-0.7.0.jar  --packages io.delta:delta-core_2.12:0.7.0 feedDeltaLakeApp.py

If you want to run PostgreSQL example, please use the following spark-submit command:

spark-submit --driver-class-path /tmp/jars/org.postgresql_postgresql-42.1.4.jar  --packages org.postgresql:postgresql:42.1.4 appendDataJdbcPrimaryKeyApp.py

Running the lab in Scala

Prerequisites:

You will need:

  • git.
  • Apache Spark (please refer Appendix P - "Spark in production: installation and a few tips").
  1. Clone this project
git clone https://github.com/jgperrin/net.jgp.books.spark.ch17

  1. cd net.jgp.books.spark.ch17

  2. Package application using sbt command

 sbt clean assembly
  1. Run Spark/Scala application using spark-submit command as shown below:
spark-submit --class net.jgp.books.spark.ch17.lab100_export.ExportWildfiresScalaApplication target/scala-2.12/SparkInAction2-Chapter17-assembly-1.0.0.jar

Notes:

  1. [Java] Due to renaming the packages to match more closely Java standards, this project is not in sync with the book's MEAP prior to v10 (published in April 2019).
  2. [Scala, Python] As of MEAP v14, we have introduced Scala and Python examples (published in October 2019).

Follow me on Twitter to get updates about the book and Apache Spark: @jgperrin. Join the book's community on Facebook or in Manning's community site.