Dean Wampler
deanwampler@gmail.com
@deanwampler
This tutorial demonstrates how to write and run Apache Spark applications using Scala with some SQL. I also teach a little Scala as we go, but if you already know Spark and you are more interested in learning just enough Scala for Spark programming, see my other tutorial Just Enough Scala for Spark.
You can run the examples and exercises several ways:
- Notebooks, like Jupyter - The easiest way, especially for data scientists accustomed to notebooks.
- In an IDE, like IntelliJ - Familiar for developers.
- At the terminal prompt using the build tool SBT.
This tutorial is mostly about learning Spark, but I teach you a little Scala as we go. If you are more interested in learning just enough Scala for Spark programming, see my new tutorial Just Enough Scala for Spark.
Notes:
- The current version of Spark used is 2.3.X, which is a bit old. (TODO!)
- While the notebook approach is the easiest way to use this tutorial to learn Spark, the IDE and SBT options show details for creating Spark applications, i.e., writing executable programs you build and run, as well as examples that use the interactive Spark Shell.
I'm grateful that several people have provided feedback, issue reports, and pull requests. In particular:
Before describing the different ways to work with the tutorial, if you're having problems, use the Gitter chat room to ask for help. You can also use the new Discussions feature for the GitHub repo. If you're reasonably certain you've found a bug, post an issue to the GitHub repo. Pull requests are welcome, too!!
Let's get started...
Begin by cloning or downloading the tutorial GitHub project github.com/deanwampler/spark-scala-tutorial.
Now Pick the way you want to work through the tutorial:
The easiest way to work with this tutorial is to use a Docker image that combines the popular Jupyter notebook environment with all the tools you need to run Spark, including the Scala language. It's called the all-spark-notebook. It bundles Apache Toree to provide Spark and Scala access. The webpage for this Docker image discusses useful information like using Python as well as Scala, user authentication topics, running your Spark jobs on clusters, rather than local mode, etc.
There are other notebook options you might investigate for your needs:
Open source:
- Polynote - A cross-language notebook environment with built-in Scala support. Developed by Netflix.
- Jupyter + BeakerX - a powerful set of extensions for Jupyter.
- Zeppelin - a popular tool in big data environments
Commercial:
- Databricks - a feature-rich, commercial, cloud-based service
If you need to install Docker, follow the installation instructions at docker.com (the community edition is sufficient).
Now we'll run the docker image. It's important to follow the next steps carefully. We're going to mount two local directories inside the running container, one for the data we want to use so and one for the notebooks.
- Open a terminal or command window
- Change to the directory where you expanded the tutorial project or cloned the repo
- To download and run the Docker image, run the following command:
run.sh(MacOS and Linux) orrun.bat(Windows)
The MacOS and Linux run.sh command executes this command:
docker run -it --rm \
-p 8888:8888 -p 4040:4040 \
--cpus=2.0 --memory=2000M \
-v "$PWD/data":/home/jovyan/data \
-v "$PWD/notebooks":/home/jovyan/notebooks \
"$@" \
jupyter/all-spark-notebookThe Windows run.bat command is similar, but uses Windows conventions.
The --cpus=... --memory=... arguments were added because the notebook "kernel" is prone to crashing with the default values. Edit to taste. Also, it will help to keep only one notebook (other than the Introduction) open at a time.
The -v PATH:/home/jovyan/dir tells Docker to mount the dir directory under your current working directory, so it's available as /home/jovyan/dir inside the container. This is essential to provide access to the tutorial data and notebooks. When you open the notebook UI (discussed shortly), you'll see these folders listed.
Note: On Windows, you may get the following error: C:\Program Files\Docker\Docker\Resources\bin\docker.exe: Error response from daemon: D: drive is not shared. Please share it in Docker for Windows Settings." If so, do the following. On your tray, next to your clock, right-click on Docker, then click on Settings. You'll see the Shared Drives. Mark your drive and hit apply. See this Docker forum thread for more tips.
The -p 8888:8888 -p 4040:4040 arguments tells Docker to "tunnel" ports 8888 and 4040 out of the container to your local environment, so you can get to the Jupyter UI at port 8888 and the Spark driver UI at 4040.
You should see output similar to the following:
Unable to find image 'jupyter/all-spark-notebook:latest' locally
latest: Pulling from jupyter/all-spark-notebook
e0a742c2abfd: Pull complete
...
ed25ef62a9dd: Pull complete
Digest: sha256:...
Status: Downloaded newer image for jupyter/all-spark-notebook:latest
Execute the command: jupyter notebook
...
[I 19:08:15.017 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 19:08:15.019 NotebookApp]
Copy/paste this URL into your browser when you connect for the first time,
to login with a token:
http://localhost:8888/?token=...Now copy and paste the URL shown in a browser window. (Use command+click in your terminal window on MacOS.)
Warning: When you quit the Docker container at the end of the tutorial, all your changes will be lost, unless they are in the
dataandnotebooksdirectories that we mounted! To save notebooks you defined in other locations, export them using the File > Download as > Notebook menu item in toolbar.
In the Jupyter UI, you should see three folders, data, notebooks, and work. The first two are the folders we mounted. The data we'll use is in the data folder. The notebooks we'll use are... you get the idea.
Open the notebooks folder and click the link for 00_Intro.ipynb.
It opens in a new browser tab. It may take several seconds to load.
Tip: If the new tab fails to open or the notebook fails to load as shown, check the terminal window where you started Jupyter. Are there any error messages?
If you're new to Jupyter, try Help > User Interface Tour to learn how to use Jupyter. At a minimum, you need to new that the content is organized into cells. You can navigate with the up and down arrows or clicks. When you come to a cell with code, either click the run button in the toolbar or use shift+return to execute the code.
Read through the Introduction notebook, then navigate to the examples using the table near the bottom. I've set up the table so that clicking each link opens a new browser tab.
The tutorial is also set up as a using the build tool SBT. The popular IDEs, like IntelliJ with the Scala plugin (required) and Eclipse with Scala, can import an SBT project and automatically create an IDE project from it.
Once imported, you can run the Spark job examples as regular applications. There are some examples implemented as scripts that need to be run using the Spark Shell or the SBT console. The tutorial goes into the details.
You are now ready to go through the tutorial.
Using SBT in a terminal is a good approach if you prefer to use a code editor like Emacs, Vim, or SublimeText. You'll need to install SBT, but not Scala or Spark. Those dependencies will be resolved when you build the software.
Start the sbt console, then build the code, where the sbt:spark-scala-tutorial> is the prompt I've configured for the project. Running test compiles the code and runs the tests, while package creates a jar file of the compiled code and configuration files:
$ sbt
...
sbt:spark-scala-tutorial> test
...
sbt:spark-scala-tutorial> package
...
sbt:spark-scala-tutorial>You are now ready to go through the tutorial.
To learn more, see the following resources:
- Lightbend's Fast Data Platform - a curated, fully-supported distribution of open-source streaming and microservice tools, like Spark, Kafka, HDFS, Akka Streams, etc.
- The Apache Spark website.
- Talks from the Spark Summit conferences.
- Learning Spark, an excellent introduction from O'Reilly, if now a bit dated.
Thank you for working through this tutorial. Feedback and pull requests are welcome.