some question
panfengfeng opened this issue · 1 comments
Hi kay,
I read your paper which is presented in NSDI 2015, and I have some question about you method. The paper says that you add some instrumentation to track the blocked time about the disk and network, and my question is what's level of the instrumentation? Dose they just track the spark or HDFS read and write operations just like taskmetrics, or they track the real read and write operations from disk or network?
Thank you very much!
The instrumentation tracked the Spark read and write operations, and most of this instrumentation is now in Spark (e.g., the fetch wait time in Spark measures time blocked on the network). All of our numbers are an upper bound on the time spent blocked because, as you alluded to, they typically involve a small amount of CPU time in addition to the network or disk time.