crash on one of our instances -- source sequence is illegal/malformed utf-8
Opened this issue · 4 comments
SteveDevOps commented
any ideas? It loops through rescue, never resolves, and I have to manually stop it.
[16-03-03 13:28:05] listening on http#0.0.0.0:4567
[16-03-03 13:28:15] !!! Redmon has shit the bed, restarting... source sequence is illegal/malformed utf-8
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon/worker.rb:27:in `to_json'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon/worker.rb:27:in `stats'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon/worker.rb:13:in `record_stats'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon/worker.rb:7:in `block in run!'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/eventmachine-1.0.9.1/lib/em/timers.rb:56:in `fire'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in `call'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in `run_machine'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:20:in `start_em'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:9:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/bin/redmon:78:in `<top (required)>'^C
[root@evest-redis-1 redmon]# service redmon_4567 stop
SteveDevOps commented
we traced through your code and found where it's trying to load stats json from the associated redis key which we identified as:
redmon:redis:<ip address>:6379:stats
We then attempted to clear it out from the redis-cli
del redmon:redis:<ip address>:6379:stats
and restart redmon, but ui fails to load and we re-enter this crash cycle. We are running redmon standalone, no integrations. Other things to try?
SteveDevOps commented
I have been able to determine that the cause of the break in run is slowlog contents. There are some HMGETs included in the slowlog get and these end with (the following for example) :
835510:516818
839761:520340
843986:524044
654885:20391
... (995 more arguments)
which I believe is what is breaking the json convert. For the moment we are running without slowlog data but would like to know a workaround?
steelThread commented
@SteveDevOps - I haven't really touched this code for a couple of years. I can take a look this weekend and get back with you. Thanks for reporting this.
SteveDevOps commented
Greatly appreciated.. let me know. Thx!