AbsaOSS/pramen

Log the list of executor nodes

Closed this issue · 0 comments

Background

This helps debugging issues.

Feature

Log the list of executor nodes

Proposed Solution

The sure way of getting this list takes time:

    val sc = spark.sparkContext

    val data = sc.parallelize(1 to 10000000).repartition(sc.defaultParallelism)
    data.mapPartitions { _ => Iterable(java.net.InetAddress.getLocalHost.getHostName).iterator }.collect().distinct.sorted

so it should be configured and turned off by default:

log.executor.nodes = false