/spark-on-docker

Primary LanguageJupyter Notebook

Spark on Docker

The project was featured on an article at Sharek.dev tech blog

Introduction

This project gives you an Apache Spark cluster in standalone mode with a JupyterLab interface built on top of Docker. Learn Apache Spark through its Scala and Python (PySpark) by running the Jupyter notebooks with examples on how to read, process and write data.

jupyterlab-latest-version spark-latest-version spark-scala-api spark-pyspark-api

TL;DR

build.sh
docker-compose up

Quick Start

Cluster overview

Application URL Description
JupyterLab localhost:8888 Cluster interface with built-in Jupyter notebooks
Spark Driver localhost:4040 Spark Driver web ui
Spark Master localhost:8080 Spark Master node
Spark Worker I localhost:8081 Spark Worker node with 1 core and 512m of memory (default)
Spark Worker II localhost:8082 Spark Worker node with 1 core and 512m of memory (default)

Prerequisites

Download from Docker Hub (easier)

  1. Download the docker compose file;
curl -LO https://raw.githubusercontent.com/cluster-apps-on-docker/spark-standalone-cluster-on-docker/master/docker-compose.yml
  1. Edit the docker compose file with your favorite tech stack version, check apps supported versions;
  2. Start the cluster;
docker-compose up
  1. Run Apache Spark code using the provided Jupyter notebooks with Scala and PySpark examples;
  2. Stop the cluster by typing ctrl+c on the terminal;
  3. Run step 3 to restart the cluster.

Build from your local machine

Note: Local build is currently only supported on Linux OS distributions.

  1. Download the source code or clone the repository;
  2. Edit the build.yml file with your favorite tech stack version;
  3. Match those version on the docker compose file;
  4. Build up the images;
chmod +x build.sh ; ./build.sh
  1. Start the cluster;
docker-compose up
  1. Run Apache Spark code using the provided Jupyter notebooks with Scala, PySpark and SparkR examples;
  2. Stop the cluster by typing ctrl+c on the terminal;

Tech Stack

  • Infra
Component Version
Docker Engine 1.13.0+
Docker Compose 1.10.0+
  • Languages and Kernels
Spark Hadoop Scala Scala Kernel Python Python Kernel
3.x 3.2 2.13.7 0.13.0 3.7.3 7.19.0
  • Apps
Component Version Docker Tag
Apache Spark 3.3.0 <spark-version>
JupyterLab 3.3.0 <jupyterlab-version>-spark-<spark-version>