MemVerge/splash

Support fast merge in the storage plugin interface.

jealous opened this issue · 0 comments

jealous commented

Allow the plugin developer to implement a fast merge which will be invoked when:

  • Encryption is disabled.
  • Compression is disabled or the compression codec supports concatenation of serialized streams.
  • The spark.shuffle.unsafe.fastMergeEnabled option is true.
  • The plugin supports fast merge.

Note the performance of this fast merge function could seriously impact the performance of Spark SQL joins with multiple spills.