The slack bytes value reaches a level of 50% of total used memory
GoogleCodeExporter opened this issue · 5 comments
GoogleCodeExporter commented
What steps will reproduce the problem?
1. Run an application under high load
2. Observe slack_bytes value
What is the expected output? What do you see instead?
I expected a relatively low value of the slack bytes (when compared to
overall memory usage). What I get is a value like: 3.8 GB of slack memory.
Overall memory used (the size of virtual address space) is: 8 GB.
What version of the product are you using? On what operating system?
google-perftools-1.3-4.el5
x86_64
CentOS release 5.2 (Final)
Please provide any additional information below.
I have a few questions. I kindly ask for answer.
a) is the slack_bytes value completely unrecoverable for tcmalloc's use?
b) does tcmalloc periodically reorganize its internal structures to regain
any slack bytes?
c) how to minimize the slack bytes value?
Original issue reported on code.google.com by Den.J...@gmail.com
on 10 May 2010 at 12:46
GoogleCodeExporter commented
tcmalloc does give back memory to the system by default (at least on linux),
but can
do so pretty slowly. You can change this via the TCMALLOC_RELEASE_RATE
environment
variable, as discussed a bit in doc/tcmalloc.html.
Original comment by csilv...@gmail.com
on 10 May 2010 at 11:20
- Changed state: NotABug
- Added labels: Priority-Medium, Type-Defect
GoogleCodeExporter commented
...but it was entirely wrong. :-( I'm sorry about that. I was lulled by the
text in
tcmalloc.html, but I think it's incorrect. (I'm looking into that more now.)
slack_bytes is actually bytes on the freelist -- either bytes reserved by
tcmalloc or
bytes given back to the OS. You can tell the difference by calling
MallocExtension::instance()->GetStats() and printing the results. If you see
lots of
memory in 'unmapped in page heap', then that's already been released to the
system
and you're ok -- the app is using virtual memory, but not physical memory.
That's
totally normal, and shouldn't cause any problems.
If the memory is 'free in page heap', then I think you'll just want to call
MallocExtension::instance()->ReleaseFreeMemory() periodically to return that
memory
back to the system.
To reiterate: I now think this data does *not* mean you have memory
fragmentation.
Try these other things first. Also, see what 'top' says as to how much
physical (not
virtual) memory your app is using. If it's a low number, you're fine.
I hope this makes sense.
Original comment by csilv...@gmail.com
on 12 May 2010 at 8:21
GoogleCodeExporter commented
Is the slack_bytes value an effect of not giving the allocated memory back to
the
system, or an effect of memory fragmentation (which should be harder to "fix")?
The slack bytes definition is as follows:
tcmalloc.slack_bytes A measure of memory fragmentation (how much memory is
reserved
by TCMalloc but unlikely to ever be able to serve an allocation request).
Original comment by pet.kh...@gmail.com
on 11 May 2010 at 7:16
GoogleCodeExporter commented
You're absolutely right -- I apologize for the mixup. I'll take another shot
at your
questions:
} is the slack_bytes value completely unrecoverable for tcmalloc's use?
Yes.
} does tcmalloc periodically reorganize its internal structures to regain
any slack bytes?
No, it doesn't reorganize. If enough memory is freed by the application,
though, it
may result in pages 'defragmenting'.
} how to minimize the slack bytes value?
Slack bytes are typically caused by 'normal' fragmentation, so if you can write
the
application in a way that it doesn't keep allocating and freeing memory, that
can
help. One way to do this -- it may be too much work for your particular case
-- is
to use an arena allocator when possible. (We have one you might be able to
make use
of in code.google.com/p/google-ctemplate, if you'd like -- it's in
src/base/arena.h.
It was written particularly for the ctemplate project, but is intended to be more
generally applicable. There are probably other c++ arena allocators around as
well.)
Also, I think it hurts a little when memory allocated by thread X is freed by a
different thread Y. I wouldn't go out of your way to restructure your code,
but if
it's easy to keep allocations/deallocations on one thread, I'd recommend it.
Original comment by csilv...@gmail.com
on 12 May 2010 at 4:01
GoogleCodeExporter commented
Thank you very much for the answer. It covers all the doubts that I had.
Original comment by Den.J...@gmail.com
on 12 May 2010 at 12:00