~/.rbot/registry_tc gets huge!
Closed this issue · 10 comments
Is it normal for old rbot's ~/.rbot/registry_tc gets this big after several years of usages?
~/.rbot/registry_tc$ ls -all
total 378364
drwxr-xr-x 6 ant ant 4096 Jan 27 2018 .
drwxr-xr-x 12 ant ant 4096 Apr 2 00:51 ..
-rw-r--r-- 1 ant ant 135680 Nov 20 2018 aliceplugin.tdb
-rw-r--r-- 1 ant ant 547072 Apr 2 00:49 authmodule.tdb
-rw-r--r-- 1 ant ant 135680 Apr 2 00:49 autoop.tdb
-rw-r--r-- 1 ant ant 136960 Apr 2 00:49 azgameplugin.tdb
-rw-r--r-- 1 ant ant 135680 Apr 2 00:49 bansplugin.tdb
-rw-r--r-- 1 ant ant 135680 May 7 2017 coinplugin.tdb
-rw-r--r-- 1 ant ant 2501888 Jul 1 2016 imdbngplugin.tdb
-rw-r--r-- 1 ant ant 140288 Feb 21 15:03 junkyardplugin.tdb
-rw-r--r-- 1 ant ant 135680 Apr 2 00:49 justinplugin.tdb
-rw-r--r-- 1 ant ant 341760 Apr 2 00:49 karmaplugin.tdb
drwxr-xr-x 2 ant ant 4096 Dec 28 2015 markovplugin
-rw-r--r-- 1 ant ant 1003008 Apr 2 00:49 noteplugin.tdb
-rw-r--r-- 1 ant ant 141568 Apr 2 00:49 pollplugin.tdb
-rw-r--r-- 1 ant ant 135680 Apr 2 00:49 reactionplugin.tdb
-rw-r--r-- 1 ant ant 136192 Apr 2 00:49 rouletteplugin.tdb
-rw-r--r-- 1 ant ant 132436992 Apr 2 00:49 rssfeedsplugin.tdb
-rw-r--r-- 1 ant ant 135680 Apr 2 00:49 scriptplugin.tdb
-rw-r--r-- 1 ant ant 1564928 Apr 2 00:49 seenplugin.tdb
-rw-r--r-- 1 ant ant 135680 Nov 6 2018 tictactoeplugin.tdb
-rw-r--r-- 1 ant ant 135680 Apr 2 2018 timeplugin.tdb
drwxr-xr-x 2 ant ant 4096 Jan 31 2016 translatorplugin
drwxr-xr-x 3 ant ant 4096 Aug 22 2018 unoplugin
-rw-r--r-- 1 ant ant 246905856 Apr 2 00:49 urlplugin.tdb
drwxr-xr-x 2 ant ant 4096 Jan 27 2018 userdatamodule
-rw-r--r-- 1 ant ant 136192 Jan 18 11:11 weatherplugin.tdb
-rw-r--r-- 1 ant ant 135680 Apr 2 00:49 whereplugin.tdb
~/.rbot/registry_tc$ du -h
288K ./unoplugin/#games
292K ./unoplugin
140K ./translatorplugin
150M ./markovplugin
140K ./userdatamodule
520M .
Long timer nice :D This is pretty normal I think especially with the markov and rss plugins which constantly write to the registry. The behavior and a fix (better make a backup before) is also explained here:
https://github.com/ruby-rbot/rbot/wiki/RegistryAdapters#database-size--optimization
I tried its manual solution, and it seems to work nicely on my remote friend's AMD Ryzen's Fedora v31 box (still waiting for my 12 years old 64-bit Debian Jessie v8 box to finish after 20 minutes with its quad-core CPU being hogged by tcbmgr process due to my markovplugin directory with its v2* files):
$ du -h
16K ./quizplugin/#linux
24K ./quizplugin
16K ./userdatamodule
892K ./markovplugin
24K ./hangmanplugin/#linux
32K ./hangmanplugin
12K ./unoplugin/#bots
20K ./unoplugin/#games
40K ./unoplugin
16K ./keywords
8.0K ./translatorplugin
2.7M .
$ ls -alh
total 1.8M
drwx------. 9 ant ant 4.0K Apr 9 12:36 .
drwx------. 12 ant ant 4.0K Apr 2 01:10 ..
-rw-r--r--. 1 ant ant 5.3K Apr 9 12:36 aliceplugin.tdb
-rw-r--r--. 1 ant ant 63K Apr 9 12:36 authmodule.tdb
-rw-r--r--. 1 ant ant 4.8K Apr 9 12:36 autoop.tdb
-rw-r--r--. 1 ant ant 6.3K Apr 9 12:36 azgameplugin.tdb
-rw-r--r--. 1 ant ant 4.8K Apr 9 12:36 bansplugin.tdb
-rw-r--r--. 1 ant ant 4.8K Apr 9 12:36 coinplugin.tdb
-rw-r--r--. 1 ant ant 4.8K Apr 9 12:36 greeterplugin.tdb
drwx------. 3 ant ant 4.0K Aug 17 2013 hangmanplugin
-rw-r--r--. 1 ant ant 4.8K Apr 9 12:36 hl2plugin.tdb
-rw-r--r--. 1 ant ant 4.8K Apr 9 12:36 horoplugin.tdb
-rw-r--r--. 1 ant ant 6.8K Apr 9 12:36 junkyardplugin.tdb
-rw-r--r--. 1 ant ant 4.8K Apr 9 12:36 justinplugin.tdb
-rw-r--r--. 1 ant ant 44K Apr 9 12:36 karmaplugin.tdb
drwx------. 2 ant ant 4.0K Apr 9 12:36 keywords
-rw-r--r--. 1 ant ant 5.3K Apr 9 12:36 loveletterplugin.tdb
drwx------. 2 ant ant 4.0K Apr 9 12:36 markovplugin
-rw-r--r--. 1 ant ant 14K Apr 9 12:36 noteplugin.tdb
-rw-r--r--. 1 ant ant 4.8K Apr 9 12:36 pointsplugin.tdb
-rw-r--r--. 1 ant ant 4.8K Apr 9 12:36 pollplugin.tdb
drwx------. 3 ant ant 4.0K Aug 13 2013 quizplugin
-rw-r--r--. 1 ant ant 5.3K Apr 9 12:36 reactionplugin.tdb
-rw-r--r--. 1 ant ant 6.5K Apr 9 12:36 rouletteplugin.tdb
-rw-r--r--. 1 ant ant 90K Apr 9 12:36 rssfeedsplugin.tdb
-rw-r--r--. 1 ant ant 4.8K Apr 9 12:36 scriptplugin.tdb
-rw-r--r--. 1 ant ant 244K Apr 9 12:36 seenplugin.tdb
-rw-r--r--. 1 ant ant 4.8K Apr 9 12:36 tictactoeplugin.tdb
-rw-r--r--. 1 ant ant 4.8K Apr 9 12:36 timeplugin.tdb
drwx------. 2 ant ant 4.0K Jun 2 2013 translatorplugin
drwx------. 4 ant ant 4.0K Apr 1 20:20 unoplugin
-rw-r--r--. 1 ant ant 1.1M Apr 9 12:36 urlplugin.tdb
drwx------. 2 ant ant 4.0K Apr 9 12:36 userdatamodule
-rw-r--r--. 1 ant ant 5.3K Apr 9 12:36 weatherplugin.tdb
-rw-r--r--. 1 ant ant 4.8K Apr 9 12:36 whereplugin.tdb
Strange that it doesn't do it automatically since I quit and relaunch my rbots often. I wonder why. Maybe a new bug? :/
It's still going:
$ ls -all
total 2101324
drwxr-xr-x 2 ant ant 4096 Apr 9 12:31 .
drwxr-xr-x 6 ant ant 4096 Apr 9 12:31 ..
-rw-r--r-- 1 ant ant 79006976 Apr 9 12:30 v2r.tdb
-rw-r--r-- 1 ant ant 79030016 Apr 9 12:31 v2.tdb
-rw-r--r-- 1 ant ant 1993693952 Apr 10 05:10 v2.tdb.tmp.131944
I think I need to abort this. What do you think?
It shouldn't take this long that's strange, have you tried the rbotdb tool to backup&restore instead?
Not yet. I just started it:
$ pwd
/home/ant/bin/rbot
$ ls
AUTHORS Gemfile man README.md tasks
bin Gemfile.lock msgmerge-wrapper.rb release-checklist test
ChangeLog INSTALL po REQUIREMENTS TODO
data launch_here.rb Rakefile setup.rb Usage_en.txt
Dockerfile lib rbot.gemspec tag-release
$ ruby bin/rbotdb backup -t tc
RBot Registry Backup/Restore/Migrate
[Ruby: 2.1.5 | DBM: GDBM version 1.8.3. 10/15/2002 (built Nov 16 2014 17:21:58) | BDB: - | TokyoCabinet: 1.4.48 | Daybreak: - | SQLite: -]
Using type=tc profile=/home/ant/.rbot registry=nil
Found registry types: bdb=0 tc=29 dbm=0 daybreak=0 sqlite=0
Using registry type: tc
~Reading... (this might take a moment)
$ top
top - 15:00:23 up 83 days, 2:46, 2 users, load average: 1.01, 0.62, 0.65
Tasks: 155 total, 3 running, 152 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.9 us, 0.9 sy, 0.0 ni, 96.0 id, 0.3 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 2057652 total, 643000 used, 1414652 free, 39488 buffers
KiB Swap: 3905532 total, 277020 used, 3628512 free. 342272 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
27110 ant 20 0 62384 20692 5272 R 102.3 1.0 3:03.04 ruby
13 root 20 0 0 0 0 S 19.2 0.0 1623:32 ksoftirqd/1
27280 ant 20 0 25764 2956 2480 R 6.4 0.1 0:00.01 top
1 root 20 0 177520 2088 1840 S 0.0 0.1 1:11.61 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:01.37 kthreadd
...
I wonder how long this will take. :/
It was still going and hogging my CPU so I decided to abort it. I decided to delete its markovplugin/ and urlplugin.tdb, and retried with the script method manually again in my ~/bin/rbot/:
$ ruby bin/rbotdb backup -t tc
RBot Registry Backup/Restore/Migrate
[Ruby: 2.1.5 | DBM: GDBM version 1.8.3. 10/15/2002 (built Nov 16 2014 17:21:58) | BDB: - | TokyoCabinet: 1.4.48 | Daybreak: - | SQLite: -]
Using type=tc profile=/home/ant/.rbot registry=nil
Found registry types: bdb=0 tc=26 dbm=0 daybreak=0 sqlite=0
Using registry type: tc
Read 26 registry files, with 1661 entries.
Written registry to ./backup_2020-04-10_204407.rbot
$ rm -R ~/.rbot/registry_tc
$ ruby bin/rbotdb restore -t tc -f backup_2020-04-10_204407.rbot RBot Registry Backup/Restore/Migrate
[Ruby: 2.1.5 | DBM: GDBM version 1.8.3. 10/15/2002 (built Nov 16 2014 17:21:58) | BDB: - | TokyoCabinet: 1.4.48 | Daybreak: - | SQLite: -]
Using type=tc profile=/home/ant/.rbot
Read 26 registry files from backup file.
Using registry type: tc
Restore successful!
$ du -h registry_tc/
20K registry_tc/unoplugin/#games
24K registry_tc/unoplugin
12K registry_tc/translatorplugin
12K registry_tc/userdatamodule
1.8M registry_tc/
I am going to guess my huge database files got corrupted over the times since my old PC was crashing hard a lot since last year. They probably got corrupted or something.
In your https://github.com/ruby-rbot/rbot/wiki/RegistryAdapters#database-size--optimization for its manual "find ~/.rbot/registry -iname '*.tdb' | xargs -n1 tcbmgr optimize" command, don't you mean registry_tc (missing _tc part)?
I am going to guess my huge database files got corrupted over the times since my old PC was crashing hard a lot since last year. They probably got corrupted or something.
Yeah that could be, probably need to do some experiments on the registry to see if there can be done anything, apparently the auto optimization/ auto vacuum didn't work for you, might be bugged.
don't you mean registry_tc (missing _tc part)?
yes, I fixed it
Thanks again. :)
For some reason, my rbots aren't automatically optimizing their databases. I have to run find ~/.rbot/registry_tc -iname '*.tdb' | xargs -n1 tcbmgr optimize command when they aren't running. :(