apache/celeborn

[DOC] Disable dynamic allocation shuffle tracking in spark 3.5

wForget opened this issue · 2 comments

wForget commented

Describe the intention you'd like

Similar to apache/incubator-uniffle#1490, we should also recommend users to disable dynamic allocation shuffle tracking by default in spark 3.5.

spark.dynamicAllocation.shuffleTracking.enabled false

/cc @pan3793

/assign @who-can-solve-this-bug

SteNicholas commented

@wForget, the SparkShuffleManager of Spark 3.X would log Detected {} (default is {}) is enabled, it's highly recommended to disable it when use Celeborn as Remote Shuffle Service to avoid performance degradation. when setting spark.dynamicAllocation.shuffleTracking.enabled to true.

  • 👍1
wForget commented

@wForget, the SparkShuffleManager of Spark 3.X would log Detected {} (default is {}) is enabled, it's highly recommended to disable it when use Celeborn as Remote Shuffle Service to avoid performance degradation. when setting spark.dynamicAllocation.shuffleTracking.enabled to true.

Got it, thanks.