Implement collection of data from executor to client.
Closed this issue · 1 comments
wonook commented
Our current implementation simply serializes and writes files from the executor, and the client reads the data to collect the RDD data. This should be handled more properly.
Please refer to edu.snu.nemo.compiler.frontend.spark.core.java.SparkFrontendUtils and package edu.snu.nemo.compiler.frontend.spark.transform.CollectTransform for more information.