The code contained in this repository shows how to setup a MinIO, using docker-compose, and how to create an Apache Spark job capable of read and write on it.
The Apache Spark code is going to execute in embedded mode, setting master to local, using all computing resources available.
This code can be used as a template in order to setup Spark to work along with MinIO in Scala.
This repository is partially based on this tutorial.
In order to execute this code, you are going to need:
- sbt
- Java 8+
- docker
- on Windows:
- setup winutil.exe and hadoop.dll, like here.
docker-compose up -d # starting miniosbt run