/sagecal-spark-docker-swarm

Apache Spark cluster in Docker Swarm

Apache License 2.0Apache-2.0

SageCal on Apache Spark (WIP)

About

The Docker swarm cluster with Apache Spark and HDFS filesystem to run SageCal. The setup uses Docker images to build scaleble cluster.

Included Software

Name Version
Hadoop 2.9
Spark 2.2.0
java OpenJDK-8
SageCal -

Note: The setup was only tested on Linux64 system.

Docker images

docker pull fdiblen/spark-worker-dirac docker pull fdiblen/spark-master-dirac docker pull fdiblen/hadoop

Instructions

Intallation instructions INSTALL.md

Instructions to submit a job JOBS.md