Keep track of hot keys
smadappa opened this issue · 7 comments
Need ability to figure out hot keys that are repeatedly being hit within a short duration and able to report them so that we can take corrective action.
Probably a tumbling window https://github.com/dgryski/go-topk? Topk-over-sliding-window is more difficult.
You could also sample keys to put into the topk sketch in the first place. If you put 1% of keys in, "with high probability" the resulting "topk" keys will also be hot over all the keys.
I've been thinking about this for the past couple days.
The main purpose of this is to be able to, on demand, get a bunch of keys such that we can process them separately (outside of rend) and decide what keys are hot. The implementation does not need to actually keep track of the top k on a sliding window all the time. I'm aiming for the minimal amount of work rend does at runtime to fulfill this requirement.
One possible solution is to just keep a counter (atomic int) and circular buffer where keys are placed during the servicing of the operations. This would keep the last N keys around, configurable of course. This would give quick access to very recent historical data, but has the disadvantage of being unclear exactly which keys would be new if the endpoint vending this data is accessed multiple times quickly. It also has the advantage of consistent performance whether or not the process is being observed. The keys would definitely take up more memory, which is at a premium for us.
Another solution is to have the request orchestrators call a log hook on every request that is normally a noop, but when someone connects and requests N keys, the next N are actually sent to that requester as they come in. Effectively we would be logging the keys on demand instead of all the time. The initial sketch in my head sets up a channel that all keys get sent on (when logging). The http handler would set up the channel and set the flag to send keys until it has satisfied the request. This would be exclusive, so a second http request would wait until the first has finished before it would be allowed to retrieve keys. The normal execution should be as close to noop as possible.
I should have noted that I'm leaning toward the latter solution, which doesn't keep keys in memory but instead streams them to the requester. It does have a downside that all operations would be serialized through a channel send, but that would be a known cost. Observation will slow down the system.
There are also redis protocol sniffers that will do this entirely outside of the process.
Do they also speak memcached? :) I didn't do a search yet, but I'd need to dig into some of the details on how those work to understand the impact to our live systems.
Oops, yes :)
https://github.com/bmatheny/memkeys <-- C++ rewrite of the orignal ruby at https://github.com/etsy/mctop
Since we have a solution for our current issue outside of this project's purview, I'm closing this issue as a wont-fix. We don't need rend to do this work, so we will not do this... for now. If it becomes necessary in the future I'll re-open.