Cannot load too much records in one time
Opened this issue · 3 comments
With the instruction https://github.com/droonga/presentation-droonga-meetup-1-introduction/blob/master/benchmark/README.md I tried to load all pages to a Droonga cluster, memory usage increased and never decreased. After 1000000+ records were loaded, a process completely ate the memory and the computer became frozen.
Result of top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3060 piro 20 0 2844040 2.013g 1.974g S 9.6 26.3 3:48.77 /usr/bin/ruby1.9.1 -S droonga-engine-worker --control-read-fd 13 --control-write-fd 23 --job-queue-socket-path /home/piro/droonga/000/db.3026.14628500.sock --pid-file /home/piro/droonga/000/dbdroonga-worker-2.pid --+
3062 piro 20 0 691364 131096 99352 S 0.0 1.6 0:08.13 /usr/bin/ruby1.9.1 -S droonga-engine-worker --control-read-fd 13 --control-write-fd 25 --job-queue-socket-path /home/piro/droonga/000/db.3026.14628500.sock --pid-file /home/piro/droonga/000/dbdroonga-worker-3.pid --+
3056 piro 20 0 674796 103684 72264 S 0.0 1.3 0:05.23 /usr/bin/ruby1.9.1 -S droonga-engine-worker --control-read-fd 13 --control-write-fd 19 --job-queue-socket-path /home/piro/droonga/000/db.3026.14628500.sock --pid-file /home/piro/droonga/000/dbdroonga-worker-0.pid --+
3058 piro 20 0 662288 70532 38856 S 0.0 0.9 0:01.98 /usr/bin/ruby1.9.1 -S droonga-engine-worker --control-read-fd 13 --control-write-fd 21 --job-queue-socket-path /home/piro/droonga/000/db.3026.14628500.sock --pid-file /home/piro/droonga/000/dbdroonga-worker-1.pid --+
3026 piro 20 0 170320 43632 5016 S 27.2 0.5 14:12.79 /usr/bin/ruby1.9.1 -S droonga-engine-service --listen-fd 15 --heartbeat-fd 16 --control-read-fd 11 --control-write-fd 14 --engine-name 192.168.200.4:10031/droonga
3021 piro 20 0 98328 18996 1696 S 0.0 0.2 0:01.97 /usr/bin/ruby1.9.1 /usr/local/bin/droonga-engine --host=192.168.200.4 --log-file=/home/piro/droonga/droonga-engine.log --daemon --pid-file=/home/piro/droonga/droonga-engine.pid
1092 lightdm 20 0 871820 8172 2816 S 0.0 0.1 0:02.70 /usr/sbin/unity-greeter
3024 piro 20 0 15664 5768 3312 S 0.3 0.1 0:23.42 /home/piro/droonga/serf agent -rpc-addr 192.168.200.4:7373 -node 192.168.200.4:10031/droonga -bind 192.168.200.4:7946 -event-handler droonga-engine-serf-event-handler -log-level warn -retry-join 192.168.200.254 -ret+
1229 lightdm 20 0 510184 3940 1472 S 0.0 0.0 0:00.34 /usr/lib/x86_64-linux-gnu/indicator-keyboard-service --use-gtk
In this result, the topmost item uses 26.3% memory. It increased 5% per one hour on my environment. After I stopped the loading command line (grndump and droonga-send), it didn't decrease but kept the percentage. To release its memory I had to kill the process with kill -KILL PID.
How many database size? (du -hc DB_DIR)
If database size is 2GiB over, droonga-engine process also uses 2GiB over. It is a natural behavior.
By the way, why did you use -KILL? The process didn't receive -TERM?
When I saw 98% memory usage with too large swap, the database was 10GiB or more. RAM of machines are 8GB (node1), 8GB (node2) and 6GB (node3). The 6GB RAM machine (node3) was dead at first. So, as you described, it leached to the maximum size of the on-memory database... OK I've understood.
Hmm, then, I think I have to shrink the database to benchmark it.
By the way, why did you use -KILL? The process didn't receive -TERM?
First I tried kill $(cat ~/droonga/droonga-engine.pid) but some processes were still there. So I killed rest processes with "-KILL" option to stop them certainly.
On the other hand, Groonga can load such too much records in one time simply. So I think this should be fixed by Droonga, even if we activate partitioning of databases.