Store and log timing/IO traces
PerilousApricot opened this issue · 2 comments
PerilousApricot commented
- Add (optional) per-file tracing for all the I/O requests against the file backing store. Allow this to be later extended to trace the I/O against the higher-level caching layer.
- Add (optional) timing tracing of
- Metadata deserialization
- Basket I/O
- Basket decompression
- Add config option to dump these info to the console
PerilousApricot commented
Hmm, need to somehow transmit these data from the executors back to the driver
PerilousApricot commented
The logging is done and implemented for both the single-process and distributed cases, and is used for unit testing. Exposing the data in production can be explored later