/async-shuffle-upload

Asynchronously replicates Spark shuffle files in a distributed storage layer.

Primary LanguageJavaApache License 2.0Apache-2.0

Spark Async Shuffle Upload Plugin

An implementation of SPARK-25299 that continues to write shuffle files to local disk, but asynchronously replicates shuffle files to a distributed storage layer.

More details to follow!

Gradle Tasks

./gradlew tasks - to get the list of gradle tasks

Start Developing

Run one of the following commands:

  • ./gradlew idea for IntelliJ
  • ./gradlew eclipse for Eclipse