spark-root/laurelin

Store and log timing/IO traces

PerilousApricot opened this issue · 2 comments

  • Add (optional) per-file tracing for all the I/O requests against the file backing store. Allow this to be later extended to trace the I/O against the higher-level caching layer.
  • Add (optional) timing tracing of
    • Metadata deserialization
    • Basket I/O
    • Basket decompression
  • Add config option to dump these info to the console

Hmm, need to somehow transmit these data from the executors back to the driver

The logging is done and implemented for both the single-process and distributed cases, and is used for unit testing. Exposing the data in production can be explored later