meerkat-dashboard/meerkat

error with events scan response: bufio.Scanner: token too long

n0braist opened this issue · 6 comments

n0braist commented

problem with reading huge environments. Connection to api is possoble, but the output for the eventstream seems to long:

tail -f .log/meerkat.log
2023/07/24 13:09:39 error with events scan response: bufio.Scanner: token too long
2023/07/24 13:09:39 Subscribing to event streams [CheckResult StateChange]
2023/07/24 13:09:39 Subscribed to event stream
2023/07/24 13:09:39 error with events scan response: bufio.Scanner: token too long
2023/07/24 13:09:39 Subscribing to event streams [CheckResult StateChange]
2023/07/24 13:09:39 Subscribed to event stream

uname -a
Linux 3.10.0-1160.90.1.el7.x86_64 #1 SMP Fri Mar 17 08:39:44 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
/usr/local/meerkat-si0vm4124_1/meerkat -v
2023/07/24 13:16:02 github.com/meerkat-dashboard/meerkat revision cf6c8a1 on 2023-07-20T02:06:02Z (dirty)

davekempe commented

Hey @n0braist Thanks for your report.
Can you give us more idea of a huge environment? How many service and or host objects are we talking? We may need to do some maths to figure out a tweak to make this work. I'm sure @63 will love the challenge if you can give some more info.

n0braist commented

In the moment around
22.000 hosts
430.000 services

63 commented

Hey @n0braist looking into this the error you posted it typically occurs when a single event fills up the buffer of the scanner.

Do you have any services that output a lot of performance data or check output?
How large are they (lines, metrics)?

63 commented

Hey @n0braist we have pushed some fixes to the branch 212-events, that should fix this issue.

We have tested this branch with:

  • A single service with 50 million characters of output.
  • A single service with 10000 metrics of performance data.
  • with 20000 hosts and services with the check frequency of 10 seconds which generated event bursts up to 20000 events per second to Meerkat

Meerkat handled all of these scenarios without a problem with these changes, if you would like to give it a test and let us know before we merge the branch into a release that would be great.

n0braist commented

hi, now it seems to work. but I will test it a liitle and give feedback again

sol1-matt commented

Merged into main branch, expect to see this in a 3.1 release along with a caching structure to further reduce load on Icinga.