martinsumner/leveled

Visibility of memory used within leveled

martinsumner opened this issue · 2 comments

Leveled logs a lot, but isn't very good at informing the operator about the shape of the merge tree (penciller) and the memory used by it.

There have been some memory-leak style issues in leveled, related to old OTP versions not GC'ing correctly when memory is no longer in use by SST processes. These have proven hard to troubleshoot after the fact.

Also related is making it clear what instance of leveled on a server is doing what. Tracing back through PIDs to determine this is proving to be too hard. Leveled needs to have an identifier it can place in logs to solve this.

when the pclerk is merging files down the tree - could it report on memory and shape of the level it is merging into:

e.g.
Level
Total number of files
Total memory used by processes
File with most memory used