High memory usage / RSS consumption
Closed this issue · 22 comments
We just added mod_myfixip on our Apache 2.4 configured as a reverse proxy and hosted on AWS behind an ELB with proxy-protocols enabled.
We have a strange pattern for the memory consumption when enabling this mod:
When disabling the mod we can see a much more linear use of memory/RSS consumption. I can see a apr_palloc made in line 340 but do not see where/when is the memory allocated for this is released ?
Just FYI, when removing this mod, we have the following pattern:
Many thanks in advance for your reply!
Best
Thanks for the report @flo8, the apr_palloc(c->pool, sizeof(my_ctx)) function "Allocate a block of memory from a pool", in this case, use the connection pool, in APR this is "automatically" freed after connection is closed.
I suppose that you have keepalive enabled (this extend the life of the connection pool).
@flo8 I found the problem, is related with connection hooks and keepalive. I will correct this in shortly (thanks for the report and detail -It has been very useful-).
Many thanks for your quick reply. I was actually about to report the same :) Thanks for the fix, I will try that out Monday and will let you know if this fixes the problem, I think it should.
Cheers.
Sorry for my late reply, it doesn't solve the problem. I am investigating what may cause the issue to persist and if it is related to mod_myfixip or not at all. Working on it and will keep you posted when I have news. Cheers
@flo8 the problem persist in the same amount of memory consumption? or now less/better?
It seems the memory is never released actually with this version. In the graph below, you can see that memory increases with time, after a while I launched a load test and memory increases dramatically but is never released obviously. Any idea ?
In this case, you say that... this versión is worse?
Indeed :)
ok, time to fix! I will try reproduce your environ...
- Can you describe me: what tools, command/parameters you used for metrics and load?
- What versions are you using?: apache 2.4.x? ubuntu 14.04? centos?
- Config: mpm_fork? mpm_worker? how many threads/process?
- Light hardware description: how many memory in the server?
Many thanks in advance.
Sure, you can find the link on serverfault here that describes the environment:
http://serverfault.com/questions/704569/memory-leak-on-apache
To get my graph, I used the Perl script below that appends memory usage into a file every minute.
#!/usr/bin/perl
open(FILE,">rss.txt");
while(1) {
my $total_rss = 0;
my @rss = `ps -aux | grep apache2 | awk -F" " '{print \$6}'`;
foreach(@rss) { $total_rss += $_; }
print $total_rss."\n";
print FILE $total_rss."\n";
sleep(60);
}
ok, about past...
- how do you do with load? apache bench (what load/parameters)?
- how many memory have this server? is a test environ or production?
more questions:
5. In serverfault you say that when you do "reload" (graceful), memory is not released, but in this environ you use the "roadrunner2 module", instead of myfixip. With myfixip, memory is released with a reload/graceful?
6. the problem with semaphores that you say in serverfault,... was solved or is mixed in our equation?
- I did a script that simulates kind of real trafic (access different URLs). It is a script I wrote that simply access several URLs, nothing fancy.
- The server has 4GB of memory and is a test server
- Yes I did use roadrunner2 module with which I had several problems, since then I switched to myfixip. Switching to myfixip allows to recover RSS after a Apache graceful restart as you can see below. The last drop in RSS is due to a apache graceful restart.
- Yes my problem is linked to RSS which are increasing and never released. Actually the script (and graph) above are showing how RSS evolve with time. At some points they should be released but they're not.
Great, thanks! Will test this right now. I will let this run for a few hours to see how RSS evolve and will keep you posted tomorrow first thing!
Good news, the RSS consumption fell dramatically. You can see on the graph below that when I applied the latest version (framed in red) the RSS consumption returned to a normal level. However, on the right graph, I made a zoom and I can still see that RSS consumption is raising over the time which means something somewhere is not released. I think under a heavier load this would be problematic but I don't know if this is related to mod_myfixip yet.
I will let the load test continue today to see the pattern over the day to see if RSS are released at some point.
Good, i will continue the review of code, thanks
Here is a quick update:
Memory is still increasing with time. Maybe settings a MaxRequestsPerChild with a low value would help ?
It also seems that a apache graceful restart helped to recover most of the memory but not all, I don't know if this is normal...
One thing I can't explain is why there was a sudden surge un memory at some point while my test script is constant in the number of requests it sends.
The good news is that with the last commit, it seems the memory is increasing much less than it used before. When you compare with the graph on the left, the result is pretty impressive!
I made a comparative, Apache 2.4 / Ubuntu 14.04 (modules: access_compat alias auth_basic authn_core authn_file authz_core authz_host authz_user autoindex dir env headers mime mpm_worker proxy proxy_http setenvif socache_shmcb ssl).
Config with (to use only 1 process/thread and infinite Keepalives -for trace memory leak-):
KeepAlive On
MaxKeepAliveRequests 0 ***
ServerLimit 1
StartServers 1
MinSpareThreads 1
MaxSpareThreads 1
ThreadLimit 1
ThreadsPerChild 1
MaxRequestWorkers 1
MaxConnectionsPerChild 0
MaxMemFree 1 ***
Load:
ab -k -c 1 -n 200000 http://server/test.txt
Metric:
#!/bin/bash
F=/tmp/apache.rss.txt
W=$(ps ax -H | awk '/bin\/apache/ { P=$1 } END { print P }')
echo Catch: $W
ps ax -H | grep "bin\/apache"
> ${F}
while [ "$(grep "^VmRSS:" /proc/${W}/status >> ${F} 2>/dev/null && echo 1)" ]; do
sleep 5
doneRSS usage with myfixip disabled/enabled:
| T1-Disabled | T2-Enabled | T3-KA-100 | T4-KA-1000 |
|---|---|---|---|
| 2816 | 2836 | 2840 | 2844 |
| 2816 | 2836 | 2840 | 2844 |
| 3660 | 3552 | 2848 | 2912 |
| 4632 | 4512 | 2848 | 2920 |
| 5576 | 5460 | 2848 | 2928 |
| 6536 | 6388 | 2848 | 2924 |
| 7528 | 7324 | 2860 | 2916 |
| 8500 | 8256 | 2848 | 2908 |
| 9492 | 9212 | 2848 | 2892 |
| 10452 | 10124 | 2848 | 2908 |
| 11420 | 11056 | 2860 | 2904 |
| 12400 | 11996 | 2860 | 2916 |
| 13364 | 12932 | 2864 | 2912 |
| 14352 | 13864 | 2848 | 2920 |
| 15324 | 14824 | 2848 | 2912 |
| 16292 | 15748 | 2864 | 2912 |
| 17268 | 16672 | 2848 | 2912 |
| 18228 | 17600 | 2848 | 2904 |
| 2816 | 18516 | 2848 | 2908 |
| 2816 | 2836 | 2840 | 2844 |
- T1 = myfixip is disabled (not loaded), MaxKeepAliveRequests=0 (unlimited)
- T2 = myfixip is enabled (and process PROXY header), MaxKeepAliveRequests=0
- T3 = myfixip is enabled (and process PROXY header), MaxKeepAliveRequests=100
- T4 = myfixip is enabled (and process PROXY header), MaxKeepAliveRequests=1000
During keepalive requests, memory is holded in connection scope, instead of request scope (I don't know by who -but no by myfixip-; seem an internal Apache issue). Setting a relatively mid value for MaxKeepAliveRequests looks a good/conservative option.
@flo8 You think that this bug can be marked as resolved?
Many thanks for the insightful info. Will look deeper into it. For the time being we can close this issue. Cheers.