Spark 2.4.7 stand alone docker image
This is a docker image of Apache Spark
Package | Version |
---|---|
python3 | 3.8.5 |
docker | 20.10.2 |
docker-compose | 1.27.4 |
spark | 2.4.7 |
This image can be used to run using the docker-compose file
- Install docker-compose.
- Run
docker-compose up
.
~/projects/personal/docker-spark-stand-alone$ sudo docker-compose up
Creating spark-worker-1 ... done
Creating spark-master ... done
Attaching to spark-worker-1, spark-master
spark-worker-1 | 2021-01-17 04:29:42,210 INFO worker.Worker: Started daemon with process name: 1@35ef8be86a36
spark-master | 2021-01-17 04:29:42,210 INFO master.Master: Started daemon with process name: 1@master
Access the UIs:
- Spark Master at spark://master:7077 (http://localhost:8080/).
- Spark Worker at 172.20.0.2:8881 (http://localhost:8081/).
Exec into a container, to run SparkScala
.
~/projects/personal/docker-spark-stand-alone$ sudo docker exec -it spark-master /bin/bash
[sudo] password for renien:
root@master:/usr/spark-2.4.7# spark-shell
2021-01-17 04:38:20,713 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = spark://master:7077, app id = app-20210117043827-0000).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.7
/_/
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_271)
Type in expressions to have them evaluated.
Type :help for more information.
scala>
SparkUI:
Docker Spark Standalone is released under the MIT © Renien.