[Improvement] Disable dynamic allocation shuffle tracking

[wForget]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

What would you like to be improved?

When using uniffle in Spark 3.5, I found that the executor did not exit in time when DRA was enabled.

spark.dynamicAllocation.enabled true
spark.shuffle.manager	org.apache.spark.shuffle.RssShuffleManager
spark.shuffle.service.enabled	false
spark.shuffle.sort.io.plugin.class	org.apache.spark.shuffle.RssShuffleDataIo

As mentioned in SPARK-39846, shuffle tracking is enabled by default in Spark 3.4.0.

When we disable shuffle service and enable shuffle tracking by default, the executor will only exit idle after shuffle cleanup. So we should disable shuffle tracking by default.

refer: https://github.com/apache/spark/blob/8dd395b2eabd2815982022b38a5287dae7af8b82/core/src/main/scala/org/apache/spark/scheduler/dynalloc/ExecutorMonitor.scala#L55

enable shuffle tracking (by default in spark 3.5):

after disable shuffle tracking:

How should we improve?

Disable dynamic allocation shuffle tracking by default.

Are you willing to submit PR?

• Yes I am willing to submit a PR!