Memory usage increasing over time

[weisiw]

Hi,

I have observed a behavior that the RSS of memtier_benchmark increases over time during tests on Redis. I found that for each client (connection), the RSS increases by 26.2 KB per second.

The test setup is as follows:

OS: Debian-based Linux

1. Start redis (following https://redis.io/download#from-source-code):

src/redis-server

2. Run memtier_benchmark:

memtier_benchmark -c 10 -t 10 -d 500 --test-time=600 --hide-histogram

3. Collect RSS data of memtier_benchmark (on a different tmux tab):

echo > /tmp/tmp; for i in `seq 1 300`; do echo -e "$i,`ps --no-headers -o rss -p $(pidof memtier_benchmark)`" | tee -a /tmp/tmp; sleep 1s; done

This is the plot of the RSS data for four combinations of threads/clients over the first 5 minutes:

I have checked the following:

• Valgrind shows that there is no memory leak.
• The output of ss -m shows that the socket buffer doesn't increase.

Has anyone seen this behavior?

I'll dig into source code more to see whether I can find anything. But appreciated it if anyone has an answer on top of head. Thanks!

[chkinglee]

Me too. The memory of memtier_benchmark continues to grow until OOM

[jingjunLi]

I integrated jeprof into memtier_benchmark and the heap profile dumps as follows:

Each Client has a run_stats m_stats, every time the Client processes a request, it calls the following API to update the run_stats.

update_set_op
update_get_op
update_arbitrary_op

Memtier_benchmark use one_second_stats in run_stats to save SET, GET, WAIT, arbitrary commands (bytes, ops, hits, and other information) for each second.
In order to count the latency histogram of each command, each command needs to declare a safe_hdr_histogram.
When using 10 threads and 10 clients, 100 one_second_stats will be generated per second.

There is a question here: why does memtier_benchmark maintain one_second_stats? We can always update m_cur_stats, do not push it to m_stats every second. (We can change some variables from unsigned int to unsigned long long int.)

The heap profile is here memtier_benchmark.pdf

[YaacovHazan]

@weisiw @jingjunLi thanks for open this issue and help to narrow it. @filipecosta90 will take a look and update.

[Jiaget]

@weisiw @jingjunLi thanks for open this issue and help to narrow it. @filipecosta90 will take a look and update.

I meet the problem too, but, what should I do if I still wanna use this benchmark tool?

[filipecosta90]

@weisiw @jingjunLi thanks for open this issue and help to narrow it. @filipecosta90 will take a look and update.

I meet the problem too, but, what should I do if I still wanna use this benchmark tool?

@Jiaget we will provide a fix for this during this week. In the meantime, I suggest you use v1.3.0 sable release.

[ictmalili]

Meet this problem too in the current mainline branch.

[yossigo]

@filipecosta90 Do you know what is the status of this issue?

[filipecosta90]

@filipecosta90 Do you know what is the status of this issue?

I believe the best approach is to keep the computed percentiles per second and not the entire histograms per second. working on it tomorrow during the day.