MemVerge/splash

Splash doesn't compile with Spark 2.4.0

Closed this issue · 3 comments

weixumv commented

Compiling Splash for Spark 2.4.0 failed with the error:

[ERROR] /git/splash/src/main/scala/org/apache/spark/shuffle/SplashShuffleReader.scala:50: type mismatch;
found : Iterator[(org.apache.spark.storage.BlockId, Long)]
required: Seq[(org.apache.spark.storage.BlockId, Long)]
[ERROR] readShuffleBlocks(shuffleBlocks)
[ERROR] ^
[ERROR] one error found
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

MapOutputTracker.getMapSizesByExecutorId() in Spark 2.4.0 has changed its return value type from Seq to Iterator. The Splash code needs to updated accordingly as well.

weixumv commented

I have a simple fix for this, though it conflicts with the current support for Spark 2.3. Please create a spark-2.4 branch for this fix. Thanks.

jealous commented

Hi @weixumv ,
Thanks for composing the fix. I created a new branch spark-240. You can use this branch for PR.

jealous commented

The fix is available in 0.4.0. Close this issue.