waku-org/nwaku

chore: reduce loglevel to some too frequent or unnecessary logs

Closed this issue · 2 comments

Ivansete-status commented

Background

We are currently flooding the logs maintained by Kibana.
We need therefore to reduce such logging stress.

 > ./elasticsearch/esclean.py -f 'status.staging' -m 'waku.relay received' -i 'logstash-2024.07.*'
Cluster: status-logs-search
logstash-2024.07.01    count:        0
logstash-2024.07.02    count:        0
logstash-2024.07.03    count:  2278120
logstash-2024.07.04    count:  5718238
logstash-2024.07.05    count:  9719812
logstash-2024.07.06    count: 10760951
logstash-2024.07.07    count:  9576777
logstash-2024.07.08    count: 11279386
logstash-2024.07.09    count:  8601145
logstash-2024.07.10    count:  7697207
logstash-2024.07.11    count:  9297693
logstash-2024.07.12    count:  9380819
logstash-2024.07.13    count:  6700679
logstash-2024.07.14    count:  6148574
logstash-2024.07.15    count:  2899015

 > ./elasticsearch/esclean.py -f 'status.staging' -m 'Dialing canceled' -i 'logstash-2024.07.*'
Cluster: status-logs-search
logstash-2024.07.01    count:        0
logstash-2024.07.02    count:        0
logstash-2024.07.03    count:    33520
logstash-2024.07.04    count:    87490
logstash-2024.07.05    count:    97000
logstash-2024.07.06    count:    46753
logstash-2024.07.07    count:    39330
logstash-2024.07.08    count:    98203
logstash-2024.07.09    count:    85934
logstash-2024.07.10    count:    88185
logstash-2024.07.11    count:   112112
logstash-2024.07.12    count:   104212
logstash-2024.07.13    count:    57672
logstash-2024.07.14    count:    55794
logstash-2024.07.15    count:    59778

 > ./elasticsearch/esclean.py -m 'no subscribed peers found' -i 'logstash-2024.07.*'            
Cluster: status-logs-search
logstash-2024.07.01    count:  4741663
logstash-2024.07.02    count:  4306861
logstash-2024.07.03    count:  4897365
logstash-2024.07.04    count:  5827581
logstash-2024.07.05    count:  7076930
logstash-2024.07.06    count:  8108590
logstash-2024.07.07    count:  8376513
logstash-2024.07.08    count:  8584226
logstash-2024.07.09    count:  6457420
logstash-2024.07.10    count:  5115558
logstash-2024.07.11    count:  6081674
logstash-2024.07.12    count:  6135034
logstash-2024.07.13    count:  4408128
logstash-2024.07.14    count:  4105543
logstash-2024.07.15    count:  2073795

 > ./elasticsearch/esclean.py -m 'handling message' -i 'logstash-2024.07.*' 
Cluster: status-logs-search
logstash-2024.07.01    count: 13597903
logstash-2024.07.02    count: 12024204
logstash-2024.07.03    count: 13113962
logstash-2024.07.04    count: 13492619
logstash-2024.07.05    count: 17239747
logstash-2024.07.06    count: 16934517
logstash-2024.07.07    count: 14967833
logstash-2024.07.08    count: 18996290
logstash-2024.07.09    count: 13809013
logstash-2024.07.10    count: 13779273
logstash-2024.07.11    count: 15914194
logstash-2024.07.12    count: 16595892
logstash-2024.07.13    count: 12909336
logstash-2024.07.14    count: 12045238
logstash-2024.07.15    count:  6006728

The issue was originally raised by @jakubgs. See the following chat:
Discord comment: https://discord.com/channels/1110799176264056863/1111539372618166294/1262344976138895431

gabrielmer commented

Taking a look at it, found that:

  1. waku.relay received logs is useful
  2. Dialing canceled is spammy but we can't change it as it's a nim-libp2p log. We can propose them to change it from debug to trace level, but if that change materializes it won't enter to Waku's v0.31.0
  3. no subscribed peers found and handling message are indeed spammy and we don't need them, they are for filter and it's enough to only log when we actually attempt to send filter messages

I also looked for other frequent logs not on this list but didn't find anything else worth deleting. So I think we can start with this

gabrielmer commented

Opened also a nim-libp2p PR for the Dialing canceled log vacp2p/nim-libp2p#1153

  • ❤️1