nadenf/rtjvm_spark_optimizations

Rock The JVM - Spark Optimizations with Scala

Rock The JVM - Spark Optimizations with Scala

Master Spark optimization techniques with Scala.

• https://rockthejvm.com/p/spark-optimization
• https://github.com/rockthejvm/spark-optimization
• https://github.com/rockthejvm/spark-optimization/releases/tag/start

Certificate

Sections

1. Scala and Spark Recap
2. Spark Performance Foundations
3. Optimizing DataFrame Transformations
4. Optimizing RDD Transformations
5. Optimizing Key-Value RDDs

Setup

IntelliJ IDEA

Install IntelliJ IDEA with the Scala plugin.

• https://www.jetbrains.com/idea/

Docker

Install Docker:

• https://docs.docker.com/desktop/install/ubuntu/
• https://docs.docker.com/engine/install/ubuntu/#set-up-the-repository

Build images:

$ cd spark-cluster
$ chmod +x build-images.sh
$ ./build-images.sh

Start dockerized Spark cluster:

$ docker compose up --scale spark-worker=3

Access each container:

# List active containers
$ docker ps
# Get a shell in any container
$ docker exec -it CONTAINER_NAME bash