Log the list of executor nodes
Closed this issue · 0 comments
yruslan commented
Background
This helps debugging issues.
Feature
Log the list of executor nodes
Proposed Solution
The sure way of getting this list takes time:
val sc = spark.sparkContext
val data = sc.parallelize(1 to 10000000).repartition(sc.defaultParallelism)
data.mapPartitions { _ => Iterable(java.net.InetAddress.getLocalHost.getHostName).iterator }.collect().distinct.sorted
so it should be configured and turned off by default:
log.executor.nodes = false