The slack bytes value reaches a level of 50% of total used memory

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Run an application under high load
2. Observe slack_bytes value

What is the expected output? What do you see instead?
I expected a relatively low value of the slack bytes (when compared to 
overall memory usage). What I get is a value like: 3.8 GB of slack memory. 
Overall memory used (the size of virtual address space) is: 8 GB.

What version of the product are you using? On what operating system?
google-perftools-1.3-4.el5
x86_64
CentOS release 5.2 (Final)

Please provide any additional information below.
I have a few questions. I kindly ask for answer.
a) is the slack_bytes value completely unrecoverable for tcmalloc's use?
b) does tcmalloc periodically reorganize its internal structures to regain 
any slack bytes?
c) how to minimize the slack bytes value?

Original issue reported on code.google.com by Den.J...@gmail.com on 10 May 2010 at 12:46

[GoogleCodeExporter]

tcmalloc does give back memory to the system by default (at least on linux), 
but can 
do so pretty slowly.  You can change this via the TCMALLOC_RELEASE_RATE 
environment 
variable, as discussed a bit in doc/tcmalloc.html.  

Original comment by csilv...@gmail.com on 10 May 2010 at 11:20

• Changed state: NotABug
• Added labels: Priority-Medium, Type-Defect

[GoogleCodeExporter]

...but it was entirely wrong. :-(  I'm sorry about that.  I was lulled by the 
text in 
tcmalloc.html, but I think it's incorrect.  (I'm looking into that more now.)

slack_bytes is actually bytes on the freelist -- either bytes reserved by 
tcmalloc or 
bytes given back to the OS.  You can tell the difference by calling 
MallocExtension::instance()->GetStats() and printing the results.  If you see 
lots of 
memory in 'unmapped in page heap', then that's already been released to the 
system 
and you're ok -- the app is using virtual memory, but not physical memory.  
That's 
totally normal, and shouldn't cause any problems.

If the memory is 'free in page heap', then I think you'll just want to call 
MallocExtension::instance()->ReleaseFreeMemory() periodically to return that 
memory 
back to the system.

To reiterate: I now think this data does *not* mean you have memory 
fragmentation.  
Try these other things first.  Also, see what 'top' says as to how much 
physical (not 
virtual) memory your app is using.  If it's a low number, you're fine.

I hope this makes sense.

Original comment by csilv...@gmail.com on 12 May 2010 at 8:21

[GoogleCodeExporter]

Is the slack_bytes value an effect of not giving the allocated memory back to 
the
system, or an effect of memory fragmentation (which should be harder to "fix")?

The slack bytes definition is as follows:
tcmalloc.slack_bytes A measure of memory fragmentation (how much memory is 
reserved
by TCMalloc but unlikely to ever be able to serve an allocation request).

Original comment by pet.kh...@gmail.com on 11 May 2010 at 7:16

[GoogleCodeExporter]

You're absolutely right -- I apologize for the mixup.  I'll take another shot 
at your
questions:

} is the slack_bytes value completely unrecoverable for tcmalloc's use?

Yes.

} does tcmalloc periodically reorganize its internal structures to regain 
any slack bytes?

No, it doesn't reorganize.  If enough memory is freed by the application, 
though, it
may result in pages 'defragmenting'.

} how to minimize the slack bytes value?

Slack bytes are typically caused by 'normal' fragmentation, so if you can write 
the
application in a way that it doesn't keep allocating and freeing memory, that 
can
help.  One way to do this -- it may be too much work for your particular case 
-- is
to use an arena allocator when possible.  (We have one you might be able to 
make use
of in code.google.com/p/google-ctemplate, if you'd like -- it's in 
src/base/arena.h.
 It was written particularly for the ctemplate project, but is intended to be more
generally applicable.  There are probably other c++ arena allocators around as 
well.)

Also, I think it hurts a little when memory allocated by thread X is freed by a
different thread Y.  I wouldn't go out of your way to restructure your code, 
but if
it's easy to keep allocations/deallocations on one thread, I'd recommend it.

Original comment by csilv...@gmail.com on 12 May 2010 at 4:01

[GoogleCodeExporter]

Thank you very much for the answer. It covers all the doubts that I had.

Original comment by Den.J...@gmail.com on 12 May 2010 at 12:00