twitter/pelikan

track inner vs outer latencies

brayniac opened this issue · 0 comments

In #342 we introduce server side latency measurements that capture the end-to-end latency within the server. Per comment on that PR, it would be interesting to track latency within the storage thread. This would provide better insight into the cause of latency.

Copying comment from @thinkingfish in that PR:

If we start breaking down the processing time at different stages and/or on different threads, we should also consider the "outer" duration vs. "inner" duration. The "outer" duration could be the time it takes from writing a request to the (storage) queue to finishing reading the corresponding response; the "inner" duration is what you are measuring here. This will give us a sense of processing latency versus queuing latency. The former indicates a slow step on the storage thread, the latter indicates when we need to consider scale out the storage thread (through partitioning, etc).

This will probably require a change in the type of content passed through the queue, instead of a simple Request object, we will need a (Request, timestamp) tuple.