litespeedtech/lsmcd

LSMCD is filling up the /tmp/lsmcd folder after the latest update

exetico opened this issue · 9 comments

exetico commented

Hi

I updated lsmcd two weeks ago, and it's filling up our /tmp location every week, or so. So almost 7,6GB of data are used.

image
image
image

I've defined the logs to be located at LogFile=/tmp/lsmcd.log with warn-level, but there's no file present.

Are there a bug in the current version?

Or, is it somewhat related to the memory available on the server? I've upgraded it, so we've moved from 8GB to 16GB.

rperper commented

Yes, this does appear to be a bug. The core files are caused by a crash in the program. You can delete the core files for now.

I have a suggestion: often the crashes can be caused by insufficient memory space being allocated for the shared cache files. So I would suggest that you make two changes to your node.conf file:

  • If you don't have a Cached.MemMaxSz defined, I might suggest that you set: Cached.MemMaxSz=1000000000 This overrides the very small default cache size and allows most memory operations to succeed avoiding most crashes.
  • Set LogLevel=notice as warn won't display most messages. This should make the `/tmp/lsmcd.log' file have entries in it.
    The procedure to activate an updated set of cache files after you make the changes above, is (as root):
/usr/local/lsmcd/bin/lsmcdctrl stop
rm -rf /dev/shm/lsmcd
/usr/local/lsmcd/bin/lsmcdctrl start

If the core files continue to keep showing up, please send me your node.conf file. It might be interesting to see what the oldest core file has in it. The output of the following command would be very interesting. If the core file was named core.128594 you could run:

gdb /usr/local/lsmcd/bin/lsmcd -c /tmp/lsmcd/core.128594
bt

The output of the backtrace (bt) command would show where the problem happened.
Thanks!

Bob Perper
rperper@litespeedtech.com

exetico commented

Hi Bib

The application has now filled up 7,6GB in five hours - after the last clear.

Steps I've now done:

  1. Opened nano /usr/local/lsmcd/conf/node.conf
  2. Added Cached.MemMaxSz=1000000000 in the end of the config file.
  3. Changed LovLevel to notice

This is the config-file, as is (after the changes). I must say that I've not changed anything, during the latest update.

Repl.HeartBeatReq=30
Repl.HeartBeatRetry=3000
Repl.MaxTidPacket=2048000
Repl.GzipStream=YES
Repl.LbAddrs=127.0.0.1:12340
Repl.ListenSvrAddr=127.0.0.1:12340
REPL.DispatchAddr=127.0.0.1:5501
RepldSockPath=/tmp/repld.usock
CACHED.PRIADDR=127.0.0.1:11000

#CACHED.ADDR=127.0.0.1:11211
CACHED.ADDR=UDS:///tmp/lsmcd.sock
#default is 8, it can be bigger depending on cache data amount
Cached.Slices=8
Cached.Slice.Priority.0=100
Cached.Slice.Priority.1=100
Cached.Slice.Priority.2=100
Cached.Slice.Priority.3=100
Cached.Slice.Priority.4=100
Cached.Slice.Priority.5=100
Cached.Slice.Priority.6=100
Cached.Slice.Priority.7=100

Cached.ShmDir=/dev/shm/lsmcd
Cached.UseSasl=false
Cached.DataByUser=true
Cached.Anonymous=false
#Cached.SaslDB=/etc/sasllsmcd

##this is the global setting, no need to have per slice configuration.
User=nobody
Group=nobody
#depends CPU core
CachedProcCnt=4
CachedSockPath=/tmp/cached.usock.
TmpDir=/tmp/lsmcd
LogLevel=notice
#LogLevel=dbg_medium
LogFile=/tmp/lsmcd.log

# Adding Cache Size Setting
Cached.MemMaxSz=1000000000

I've tried to run the gdb command, but it's reporting that it's not a core-file dump. Example error from the command is: "/tmp/lsmcd/core.6416" is not a core dump:

I've now removed all the core files with sudo rm core.* in the /tmp/lsmcd folder. rm -rf /dev/shm/lsmcd are executed, too.

Let's see if the Cached.MemMaxSz does the trick. If so, I think it's very important to add some information about that specific option :-)

rperper commented

Version 1.4.20 has been released. It has a number of bug fixes in it which should, hopefully, reduce the frequency of crashes. And it will only preserve the oldest core file so that it no longer hogs disk. Please upgrade when you get a chance and let me know if it addresses your problems.

Thanks,

Bob Perper
rperper@litespeedtech.com

exetico commented

Hi rperper

I've grapped the latest version, compiled it and installed it.

Please consider the following things:

  1. Always echo "Install Done", or something like that. If I execute dist]# ./install.sh, it's returning nothing.
  2. Consider to return the correct version, if the user connects to the lsmcd-service through socat. The version command returns VERSION 1.0.0.

With that said; Thank you for your support, and a great memcache-replacement, too :-)

rperper commented

Just wanted to address your points:

  1. In the first-time install it gives you further instructions, but in the update, it just completes with no message. The next release will say "Update completed". Nice suggestion.
  2. I never liked having to report the version as 1.0.0, but that's the version of the API, not the version of the software. Part of the memcached requirements. Sigh. You can get the version by entering /usr/local/lsmcd/bin/lsmcd -v
    Thanks for your support!

Bob Perper
rperper@litespeedtech.com

  • 👍1
exetico commented

That's nice to know! Thanks. I'll save /usr/local/lsmcd/bin/lsmcd -v. Maybe you should query that after the install, or at least add is as a Update completed. You can check the current version with "/usr/local/lsmcd/bin/lsmcd -v" or so :)

rperper commented

Sorry, the version wasn't pushed correctly. Can you pick it up now. It should be correct (and have a message after the install as well).

Thanks,

Bob Perper
rperper@litespeedtech.com

exetico commented

1 - So far so good?

So far so good (without the latest update, you've requested 4 hours ago).

No issues with the tmp-folder filling up, but I'm still using the Cached.MemMaxSz=1000000000 option. I'm not sure it there's any risk related to that? Should I remove that, after the update? Or correct it, to a bit smaller value?

Let me know if something could be useful for you.

2 -It's now updated, again.

 1004  cd ~/
 1009  sudo rm -r lsmcd
 1010  git clone https://github.com/litespeedtech/lsmcd.git
 1011  cd lsmcd
 1012  ./fixtimestamp.sh
 1013  ./configure CFLAGS=" -O3" CXXFLAGS=" -O3"
 1014  make
 1015  cd dist
 1016  systemctl stop lsmcd
 1018  ./install.sh
 1019  systemctl start lsmcd
 1020  tail -5 /tmp/lsmcd.log
 1021  socat - UNIX-CONNECT:/tmp/lsmcd.sock,crnl
 1022  systemctl enable lsmcd
 1023  /usr/local/lsmcd/bin/lsmcd -v

Here's my update-steps, if anyone can use them.

The install-script reported:

# ./install.sh
Update completed

The version reports:

# /usr/local/lsmcd/bin/lsmcd -v
lsmcd server 1.4.20
rperper commented

I'd leave it at a bigger value, if you can afford the memory. Most likely the crashes were related to insufficient memory in the shared memory space and this fixes that.

Thanks,

Bob Perper
rperper@litespeedtech.com

  • 👍1